akka-cluster > multi-jvm:test-only akka.cluster.ConvergenceWithFailureDetectorPuppet
[info] * akka.cluster.ConvergenceWithFailureDetectorPuppet
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.429] [main] [EventStream(akka://ConvergenceSpec)] logger log1-TestEventListener started
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.439] [main] [EventStream(akka://ConvergenceSpec)] Default Loggers started
[JVM-Node2] [DEBUG] [06/14/2012 08:19:01.488] [main] [EventStream(akka://ConvergenceSpec)] logger log1-TestEventListener started
[JVM-Node2] [DEBUG] [06/14/2012 08:19:01.491] [main] [EventStream(akka://ConvergenceSpec)] Default Loggers started
[JVM-Node1] [INFO] [06/14/2012 08:19:01.610] [main] [ActorSystem(ConvergenceSpec)] RemoteServerStarted@akka://ConvergenceSpec@localhost:61364
[JVM-Node3] [DEBUG] [06/14/2012 08:19:01.689] [main] [EventStream(akka://ConvergenceSpec)] logger log1-TestEventListener started
[JVM-Node3] [DEBUG] [06/14/2012 08:19:01.699] [main] [EventStream(akka://ConvergenceSpec)] Default Loggers started
[JVM-Node2] [INFO] [06/14/2012 08:19:01.723] [main] [ActorSystem(ConvergenceSpec)] RemoteServerStarted@akka://ConvergenceSpec@localhost:61367
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.743] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller] received handled message GetSockAddr
[JVM-Node4] [DEBUG] [06/14/2012 08:19:01.754] [main] [EventStream(akka://ConvergenceSpec)] logger log1-TestEventListener started
[JVM-Node4] [DEBUG] [06/14/2012 08:19:01.757] [main] [EventStream(akka://ConvergenceSpec)] Default Loggers started
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.813] [ConvergenceSpec-akka.actor.default-dispatcher-4] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1] open
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.833] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] connection from /127.0.0.1:61368
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.836] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller] received handled message CreateServerFSM([id: 0x0d896a4c, /127.0.0.1:61368 => /127.0.0.1:4711])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.836] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] bound
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.836] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] connected to /127.0.0.1:4711
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.852] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Connected([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Data(None,None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.876] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] transition Connecting -> AwaitDone
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.879] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 53
[JVM-Node3] [INFO] [06/14/2012 08:19:01.900] [main] [ActorSystem(ConvergenceSpec)] RemoteServerStarted@akka://ConvergenceSpec@localhost:61371
[JVM-Node2] [DEBUG] [06/14/2012 08:19:01.904] [ConvergenceSpec-akka.actor.default-dispatcher-3] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6] open
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.909] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] connection from /127.0.0.1:61372
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.910] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller] received handled message CreateServerFSM([id: 0x459d3b3a, /127.0.0.1:61372 => /127.0.0.1:4711])
[JVM-Node2] [DEBUG] [06/14/2012 08:19:01.921] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] bound
[JVM-Node2] [DEBUG] [06/14/2012 08:19:01.922] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] connected to localhost/127.0.0.1:4711
[JVM-Node2] [DEBUG] [06/14/2012 08:19:01.948] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Connected([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Data(None,None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node2] [DEBUG] [06/14/2012 08:19:01.976] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] transition Connecting -> AwaitDone
[JVM-Node2] [DEBUG] [06/14/2012 08:19:01.977] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 54
[JVM-Node4] [INFO] [06/14/2012 08:19:01.981] [main] [ActorSystem(ConvergenceSpec)] RemoteServerStarted@akka://ConvergenceSpec@localhost:61375
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.997] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: Hello(first,akka://ConvergenceSpec@localhost:61364)
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.997] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: Hello(second,akka://ConvergenceSpec@localhost:61367)
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.998] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(Hello(first,akka://ConvergenceSpec@localhost:61364),None) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.998] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] transition Initial -> Ready
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.998] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(Hello(second,akka://ConvergenceSpec@localhost:61367),None) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.999] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] transition Initial -> Ready
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.999] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message NodeInfo(RoleName(first),akka://ConvergenceSpec@localhost:61364,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.999] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message NodeInfo(RoleName(second),akka://ConvergenceSpec@localhost:61367,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:01.999] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] processing Event(NodeInfo(RoleName(first),akka://ConvergenceSpec@localhost:61364,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]),Data(Set(),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.000] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] processing Event(NodeInfo(RoleName(second),akka://ConvergenceSpec@localhost:61367,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@localhost:61364,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]
[JVM-Node3] [DEBUG] [06/14/2012 08:19:02.073] [ConvergenceSpec-akka.actor.default-dispatcher-2] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0fc8fb4b] open
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.075] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] connection from /127.0.0.1:61376
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.076] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message CreateServerFSM([id: 0x3ecec78d, /127.0.0.1:61376 => /127.0.0.1:4711])
[JVM-Node3] [DEBUG] [06/14/2012 08:19:02.080] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711] bound
[JVM-Node3] [DEBUG] [06/14/2012 08:19:02.080] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] connected to localhost/127.0.0.1:4711
[JVM-Node3] [DEBUG] [06/14/2012 08:19:02.092] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Connected([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),Data(None,None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/14/2012 08:19:02.109] [ConvergenceSpec-akka.actor.default-dispatcher-3] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105] open
[JVM-Node3] [DEBUG] [06/14/2012 08:19:02.111] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] transition Connecting -> AwaitDone
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.111] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] connection from /127.0.0.1:61377
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.112] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message CreateServerFSM([id: 0x4e8890da, /127.0.0.1:61377 => /127.0.0.1:4711])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.113] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61376: Hello(third,akka://ConvergenceSpec@localhost:61371)
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.113] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(Hello(third,akka://ConvergenceSpec@localhost:61371),None) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.114] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] transition Initial -> Ready
[JVM-Node4] [DEBUG] [06/14/2012 08:19:02.117] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] bound
[JVM-Node3] [DEBUG] [06/14/2012 08:19:02.114] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711] written 53
[JVM-Node4] [DEBUG] [06/14/2012 08:19:02.117] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] connected to localhost/127.0.0.1:4711
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.114] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message NodeInfo(RoleName(third),akka://ConvergenceSpec@localhost:61371,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.114] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/barriers] processing Event(NodeInfo(RoleName(third),akka://ConvergenceSpec@localhost:61371,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@localhost:61364,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@localhost:61367,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]
[JVM-Node4] [DEBUG] [06/14/2012 08:19:02.129] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Connected([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Data(None,None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/14/2012 08:19:02.144] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] transition Connecting -> AwaitDone
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.145] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: Hello(fourth,akka://ConvergenceSpec@localhost:61375)
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.146] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(Hello(fourth,akka://ConvergenceSpec@localhost:61375),None) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/14/2012 08:19:02.146] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 54
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.146] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] transition Initial -> Ready
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.146] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message NodeInfo(RoleName(fourth),akka://ConvergenceSpec@localhost:61375,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.146] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/barriers] processing Event(NodeInfo(RoleName(fourth),akka://ConvergenceSpec@localhost:61375,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@localhost:61364,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@localhost:61367,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@localhost:61371,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.150] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(ToClient(Done),None) from Actor[akka://ConvergenceSpec/user/controller]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.150] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(Done),None) from Actor[akka://ConvergenceSpec/user/controller]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.150] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(Done),None) from Actor[akka://ConvergenceSpec/user/controller]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.150] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(Done),None) from Actor[akka://ConvergenceSpec/user/controller]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.152] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: Done
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.153] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Done,Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.153] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] received Done: starting test
[JVM-Node1] [DEBUG] [06/14/2012 08:19:02.153] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] transition AwaitDone -> Connected
[JVM-Node1] [INFO] [06/14/2012 08:19:02.157] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode1(akka://ConvergenceSpec)] Role [first] started
[first] ConvergenceWithFailureDetectorPuppetMultiJvmNode1:
[first] A cluster of 3 members 
[JVM-Node2] [DEBUG] [06/14/2012 08:19:02.230] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: Done
[JVM-Node2] [DEBUG] [06/14/2012 08:19:02.230] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Done,Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node2] [DEBUG] [06/14/2012 08:19:02.230] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] received Done: starting test
[JVM-Node2] [DEBUG] [06/14/2012 08:19:02.231] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] transition AwaitDone -> Connected
[JVM-Node2] [INFO] [06/14/2012 08:19:02.235] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode2(akka://ConvergenceSpec)] Role [second] started
[JVM-Node4] [DEBUG] [06/14/2012 08:19:02.243] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: Done
[JVM-Node4] [DEBUG] [06/14/2012 08:19:02.243] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Done,Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/14/2012 08:19:02.243] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] received Done: starting test
[JVM-Node4] [DEBUG] [06/14/2012 08:19:02.244] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] transition AwaitDone -> Connected
[JVM-Node4] [INFO] [06/14/2012 08:19:02.247] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode4(akka://ConvergenceSpec)] Role [fourth] started
[JVM-Node3] [DEBUG] [06/14/2012 08:19:02.258] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: Done
[JVM-Node3] [DEBUG] [06/14/2012 08:19:02.258] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Done,Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node3] [DEBUG] [06/14/2012 08:19:02.259] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] received Done: starting test
[JVM-Node3] [DEBUG] [06/14/2012 08:19:02.259] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] transition AwaitDone -> Connected
[JVM-Node3] [INFO] [06/14/2012 08:19:02.262] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode3(akka://ConvergenceSpec)] Role [third] started
[second] ConvergenceWithFailureDetectorPuppetMultiJvmNode2:
[second] A cluster of 3 members 
[fourth] ConvergenceWithFailureDetectorPuppetMultiJvmNode4:
[second] [DEBUG] [06/14/2012 08:19:02.304] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-started)
[fourth] A cluster of 3 members 
[second] [DEBUG] [06/14/2012 08:19:02.307] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-started)),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$a]
[second] [DEBUG] [06/14/2012 08:19:02.308] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 21
[first] [DEBUG] [06/14/2012 08:19:02.308] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:02.309] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(EnterBarrier(first-started),None) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:02.313] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-started)
[third] ConvergenceWithFailureDetectorPuppetMultiJvmNode3:
[first] [DEBUG] [06/14/2012 08:19:02.309] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:02.309] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-started),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]
[first] [DEBUG] [06/14/2012 08:19:02.310] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/barriers] setting timer 'Timeout'/30000 milliseconds: StateTimeout
[first] [DEBUG] [06/14/2012 08:19:02.311] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/barriers] transition Idle -> Waiting
[fourth] [DEBUG] [06/14/2012 08:19:02.315] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-started)),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$a]
[fourth] [DEBUG] [06/14/2012 08:19:02.316] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 21
[first] [DEBUG] [06/14/2012 08:19:02.316] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:02.316] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(EnterBarrier(first-started),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:02.317] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:02.317] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-started),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]
[third] A cluster of 3 members 
[first] [DEBUG] [06/14/2012 08:19:02.318] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2])))
[third] [DEBUG] [06/14/2012 08:19:02.323] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-started)
[third] [DEBUG] [06/14/2012 08:19:02.325] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-started)),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$a]
[third] [DEBUG] [06/14/2012 08:19:02.326] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711] written 21
[first] [DEBUG] [06/14/2012 08:19:02.326] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61376: EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:02.326] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(EnterBarrier(first-started),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:02.327] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:02.327] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-started),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]
[first] [DEBUG] [06/14/2012 08:19:02.327] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2])))
[first] [INFO] [06/14/2012 08:19:02.328] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - is starting up...
[first] [INFO] [06/14/2012 08:19:02.348] [main] [Node(akka://ConvergenceSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[first] [INFO] [06/14/2012 08:19:02.363] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - registering cluster JMX MBean [akka:type=Cluster]
[first] [INFO] [06/14/2012 08:19:02.364] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - has started up successfully
[first] [DEBUG] [06/14/2012 08:19:02.365] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-started)
[first] [DEBUG] [06/14/2012 08:19:02.367] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-started)),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$a]
[first] [DEBUG] [06/14/2012 08:19:02.368] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 21
[first] [DEBUG] [06/14/2012 08:19:02.368] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:02.368] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(EnterBarrier(first-started),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:02.368] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:02.368] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-started),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]
[first] [DEBUG] [06/14/2012 08:19:02.369] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2])))
[first] [DEBUG] [06/14/2012 08:19:02.370] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] canceling timer 'Timeout'
[first] [DEBUG] [06/14/2012 08:19:02.370] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(BarrierResult(first-started,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:02.370] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(ToClient(BarrierResult(first-started,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:02.370] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] transition Waiting -> Idle
[first] [DEBUG] [06/14/2012 08:19:02.370] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(BarrierResult(first-started,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[third] [DEBUG] [06/14/2012 08:19:02.370] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-started,true)
[first] [DEBUG] [06/14/2012 08:19:02.371] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(BarrierResult(first-started,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:02.371] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: BarrierResult(first-started,true)
[fourth] [DEBUG] [06/14/2012 08:19:02.371] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-started,true)
[first] [DEBUG] [06/14/2012 08:19:02.371] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-started,true),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((first-started,Actor[akka://ConvergenceSpec/temp/$a])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:02.371] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-started,true),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),Some((first-started,Actor[akka://ConvergenceSpec/temp/$a])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:02.371] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-started,true)
[second] [DEBUG] [06/14/2012 08:19:02.371] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-started,true),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((first-started,Actor[akka://ConvergenceSpec/temp/$a])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:02.371] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-started,true),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Some((first-started,Actor[akka://ConvergenceSpec/temp/$a])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:02.371] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-started
[second] [DEBUG] [06/14/2012 08:19:02.372] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-started
[third] [DEBUG] [06/14/2012 08:19:02.372] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-started
[fourth] [DEBUG] [06/14/2012 08:19:02.372] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-started
[fourth] [DEBUG] [06/14/2012 08:19:02.373] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-second-third-joined)
[fourth] [DEBUG] [06/14/2012 08:19:02.373] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-second-third-joined)),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$b]
[fourth] [DEBUG] [06/14/2012 08:19:02.373] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 33
[first] [DEBUG] [06/14/2012 08:19:02.374] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:02.374] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(EnterBarrier(first-second-third-joined),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:02.374] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:02.374] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-second-third-joined),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]
[first] [DEBUG] [06/14/2012 08:19:02.375] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] setting timer 'Timeout'/30000 milliseconds: StateTimeout
[first] [DEBUG] [06/14/2012 08:19:02.375] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] transition Idle -> Waiting
[third] [INFO] [06/14/2012 08:19:02.461] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - is starting up...
[second] [INFO] [06/14/2012 08:19:02.461] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - is starting up...
[second] [INFO] [06/14/2012 08:19:02.487] [main] [Node(akka://ConvergenceSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[third] [INFO] [06/14/2012 08:19:02.489] [main] [Node(akka://ConvergenceSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[second] [INFO] [06/14/2012 08:19:02.502] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - registering cluster JMX MBean [akka:type=Cluster]
[second] [INFO] [06/14/2012 08:19:02.503] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - has started up successfully
[second] [DEBUG] [06/14/2012 08:19:02.504] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$b]
[third] [INFO] [06/14/2012 08:19:02.505] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - registering cluster JMX MBean [akka:type=Cluster]
[second] [DEBUG] [06/14/2012 08:19:02.505] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 13
[third] [INFO] [06/14/2012 08:19:02.506] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - has started up successfully
[first] [DEBUG] [06/14/2012 08:19:02.506] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:02.506] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:02.506] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:02.507] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[third] [DEBUG] [06/14/2012 08:19:02.507] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$b]
[second] [DEBUG] [06/14/2012 08:19:02.508] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[second] [DEBUG] [06/14/2012 08:19:02.509] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$b])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:02.509] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:02.509] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61376: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:02.510] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:02.510] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:02.510] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[second] [INFO] [06/14/2012 08:19:02.510] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Trying to send JOIN to [akka://ConvergenceSpec@first] through connection [Actor[akka://ConvergenceSpec@first/system/cluster/commands]]
[third] [DEBUG] [06/14/2012 08:19:02.511] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[third] [DEBUG] [06/14/2012 08:19:02.511] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$b])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [INFO] [06/14/2012 08:19:02.513] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Trying to send JOIN to [akka://ConvergenceSpec@first] through connection [Actor[akka://ConvergenceSpec@first/system/cluster/commands]]
[second] [DEBUG] [06/14/2012 08:19:02.515] [main] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@first|localhost/127.0.0.1]
[third] [DEBUG] [06/14/2012 08:19:02.518] [main] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@first|localhost/127.0.0.1]
[second] [DEBUG] [06/14/2012 08:19:02.540] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@first
[third] [DEBUG] [06/14/2012 08:19:02.542] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@first
[second] [DEBUG] [06/14/2012 08:19:02.543] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[second] [INFO] [06/14/2012 08:19:02.545] [main] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@first
[second] [DEBUG] [06/14/2012 08:19:02.545] [main] [RemoteClient(akka://ConvergenceSpec)] Sending message Join(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/commands]
[third] [DEBUG] [06/14/2012 08:19:02.547] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[second] [DEBUG] [06/14/2012 08:19:02.550] [main] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.ClusterAction$Join]
[third] [INFO] [06/14/2012 08:19:02.548] [main] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@first
[third] [DEBUG] [06/14/2012 08:19:02.548] [main] [RemoteClient(akka://ConvergenceSpec)] Sending message Join(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@first/system/cluster/commands]
[third] [DEBUG] [06/14/2012 08:19:02.554] [main] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.ClusterAction$Join]
[first] [DEBUG] [06/14/2012 08:19:02.565] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@localhost:61379
[first] [DEBUG] [06/14/2012 08:19:02.566] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@localhost:61378
[second] [DEBUG] [06/14/2012 08:19:02.568] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@344e17f9),CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:02.568] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[second] [DEBUG] [06/14/2012 08:19:02.570] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[second] [DEBUG] [06/14/2012 08:19:02.570] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@395fd251),CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:02.571] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.571] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[third] [DEBUG] [06/14/2012 08:19:02.571] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@25861086),CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.572] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.572] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[first] [DEBUG] [06/14/2012 08:19:02.573] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[third] [DEBUG] [06/14/2012 08:19:02.574] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[third] [DEBUG] [06/14/2012 08:19:02.574] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@16e9b4bb),CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.574] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.575] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[first] [DEBUG] [06/14/2012 08:19:02.590] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:02.591] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:02.591] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[first] [DEBUG] [06/14/2012 08:19:02.591] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[first] [DEBUG] [06/14/2012 08:19:02.592] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:02.593] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:02.593] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[first] [DEBUG] [06/14/2012 08:19:02.593] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[first] [INFO] [06/14/2012 08:19:02.595] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:02.595] [ConvergenceSpec-2] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:02.596] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:02.596] [ConvergenceSpec-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:02.597] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteServerClientConnected@akka://ConvergenceSpec@first: Client[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:02.597] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteServerClientConnected@akka://ConvergenceSpec@first: Client[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:02.611] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Join(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/commands]<+{akka://ConvergenceSpec@first/system/cluster/commands} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:02.611] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Join(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/commands]<+{akka://ConvergenceSpec@first/system/cluster/commands} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [INFO] [06/14/2012 08:19:02.611] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Node [akka://ConvergenceSpec@second] is JOINING
[first] [DEBUG] [06/14/2012 08:19:02.613] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:02.614] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Node [akka://ConvergenceSpec@third] is JOINING
[first] [DEBUG] [06/14/2012 08:19:02.615] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:02.683] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:02.685] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:02.686] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:02.686] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:02.687] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:02.687] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to members not in seen table [Set(akka://ConvergenceSpec@second, akka://ConvergenceSpec@third)].
[first] [DEBUG] [06/14/2012 08:19:02.688] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:02.689] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@first -> VectorClock(Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:02.690] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.GossipEnvelope]
[first] [DEBUG] [06/14/2012 08:19:02.719] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2518, cap=2518, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.719] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1436ae83),CompositeChannelBuffer(ridx=0, widx=2518, cap=2518, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:02.720] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2518, cap=2518, components=2)
[third] [DEBUG] [06/14/2012 08:19:02.720] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[third] [DEBUG] [06/14/2012 08:19:02.721] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1494, cap=1494)
[third] [DEBUG] [06/14/2012 08:19:02.721] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.721] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[third] [DEBUG] [06/14/2012 08:19:02.722] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=1494, cap=1494)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.722] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1494, cap=1494)
[third] [DEBUG] [06/14/2012 08:19:02.753] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@first -> VectorClock(Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [INFO] [06/14/2012 08:19:02.762] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Can't establish a causal relationship between "remote" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@first -> VectorClock(Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))] and "local" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a521d6)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a521d6))] - merging them into [Gossip(overview = GossipOverview(seen = [], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))]
[third] [DEBUG] [06/14/2012 08:19:02.762] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:02.783] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:02.783] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:02.783] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.Heartbeat]
[first] [DEBUG] [06/14/2012 08:19:02.784] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.785] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:02.785] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4b8706f6),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:02.785] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:02.785] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.785] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:02.785] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:02.785] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6d4492fb),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:02.786] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:02.786] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:02.786] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:02.786] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:02.786] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.786] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:02.788] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:02.789] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:02.792] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:02.795] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:02.822] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:02.824] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@second, status = Joining)]
[third] [DEBUG] [06/14/2012 08:19:02.824] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[second] [INFO] [06/14/2012 08:19:02.825] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Leader is moving node [akka://ConvergenceSpec@second] from JOINING to UP
[third] [DEBUG] [06/14/2012 08:19:02.827] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:02.828] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:02.828] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:02.828] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:02.834] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:02.834] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:02.835] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.GossipEnvelope]
[third] [DEBUG] [06/14/2012 08:19:02.840] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)
[third] [DEBUG] [06/14/2012 08:19:02.840] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@41c7d56b),CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.840] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.840] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[first] [DEBUG] [06/14/2012 08:19:02.841] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1814, cap=1814)
[first] [DEBUG] [06/14/2012 08:19:02.841] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:02.841] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[third] [DEBUG] [06/14/2012 08:19:02.841] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:02.842] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:02.842] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:02.843] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=1814, cap=1814)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.843] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@second
[third] [DEBUG] [06/14/2012 08:19:02.843] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[third] [INFO] [06/14/2012 08:19:02.843] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@second
[third] [DEBUG] [06/14/2012 08:19:02.843] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1b78efd8),CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.844] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.843] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1814, cap=1814)
[third] [DEBUG] [06/14/2012 08:19:02.844] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:02.845] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)
[third] [DEBUG] [06/14/2012 08:19:02.846] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@12d28dfa),CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.846] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)
[second] [DEBUG] [06/14/2012 08:19:02.846] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@localhost:61380
[second] [DEBUG] [06/14/2012 08:19:02.847] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:02.847] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1856, cap=1856)
[second] [DEBUG] [06/14/2012 08:19:02.851] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:02.851] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[first] [DEBUG] [06/14/2012 08:19:02.852] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:02.853] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=1856, cap=1856)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [INFO] [06/14/2012 08:19:02.853] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:02.853] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1856, cap=1856)
[second] [DEBUG] [06/14/2012 08:19:02.854] [ConvergenceSpec-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:02.855] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteServerClientConnected@akka://ConvergenceSpec@second: Client[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:02.857] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:02.881] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:02.881] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:02.882] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:02.882] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:02.882] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:02.882] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to members not in seen table [Set(akka://ConvergenceSpec@second)].
[first] [DEBUG] [06/14/2012 08:19:02.882] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:02.882] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:02.884] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.885] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@74c12978),CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:02.885] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)
[third] [DEBUG] [06/14/2012 08:19:02.885] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[third] [DEBUG] [06/14/2012 08:19:02.886] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=788, cap=788)
[third] [DEBUG] [06/14/2012 08:19:02.886] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.886] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[third] [DEBUG] [06/14/2012 08:19:02.886] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=788, cap=788)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.886] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=788, cap=788)
[third] [DEBUG] [06/14/2012 08:19:02.890] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:02.892] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:02.903] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [INFO] [06/14/2012 08:19:02.911] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Can't establish a causal relationship between "remote" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))] and "local" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5232a)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5232a))] - merging them into [Gossip(overview = GossipOverview(seen = [], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))]
[second] [DEBUG] [06/14/2012 08:19:02.911] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:02.922] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:02.922] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:02.922] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.Heartbeat]
[second] [DEBUG] [06/14/2012 08:19:02.923] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:02.923] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:02.923] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@28996b65),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:02.923] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:02.923] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:02.924] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.923] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:02.924] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@16881cb3),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:02.924] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.924] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:02.925] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:02.925] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:02.925] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:02.925] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:02.925] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.Heartbeat]
[third] [DEBUG] [06/14/2012 08:19:02.925] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.926] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:02.926] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:02.926] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:02.926] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6243487e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.926] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:02.927] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:02.927] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:02.927] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:02.927] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:02.927] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5c0e920f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:02.928] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:02.927] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:02.927] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:02.928] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:02.928] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:02.929] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:02.929] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:02.929] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:02.929] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:02.930] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:02.930] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:02.930] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:02.930] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.020] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:03.021] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:03.022] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:03.022] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.022] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:03.023] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[third] [DEBUG] [06/14/2012 08:19:03.023] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:03.023] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:03.023] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:03.023] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:03.024] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:03.024] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.024] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:03.025] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:03.025] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.GossipEnvelope]
[third] [DEBUG] [06/14/2012 08:19:03.026] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2896, cap=2896, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.026] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@685f1ba8),CompositeChannelBuffer(ridx=0, widx=2896, cap=2896, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.026] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2896, cap=2896, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.026] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:03.027] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=848, cap=848)
[second] [DEBUG] [06/14/2012 08:19:03.027] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.027] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:03.027] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=848, cap=848)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.028] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=848, cap=848)
[second] [DEBUG] [06/14/2012 08:19:03.030] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.031] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.032] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.032] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@30cafa1),CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.032] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.033] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[third] [DEBUG] [06/14/2012 08:19:03.033] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1876, cap=1876)
[third] [DEBUG] [06/14/2012 08:19:03.033] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.033] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[third] [DEBUG] [06/14/2012 08:19:03.033] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=1876, cap=1876)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.034] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1876, cap=1876)
[third] [DEBUG] [06/14/2012 08:19:03.042] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:03.043] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:03.079] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:03.079] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:03.079] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:03.079] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to members not in seen table [Set(akka://ConvergenceSpec@second)].
[first] [DEBUG] [06/14/2012 08:19:03.080] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.080] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.080] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:03.080] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.082] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.083] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6ce7ce4c),CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.083] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.083] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:03.083] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1812, cap=1812)
[second] [DEBUG] [06/14/2012 08:19:03.084] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.084] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:03.084] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=1812, cap=1812)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.084] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1812, cap=1812)
[second] [DEBUG] [06/14/2012 08:19:03.088] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.089] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:03.178] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:03.178] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.179] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.179] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:03.179] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6699176f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.179] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.180] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.180] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.179] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:03.180] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.180] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.180] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@59b70f9e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.180] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.180] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:03.181] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.181] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:03.181] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.181] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:03.182] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:03.182] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:03.218] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:03.218] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:03.218] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:03.218] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.219] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.219] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.221] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:03.219] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:03.220] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.221] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[third] [DEBUG] [06/14/2012 08:19:03.221] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=852, cap=852)
[second] [DEBUG] [06/14/2012 08:19:03.220] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2bd9e5ad),CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.220] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.224] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2996, cap=2996)
[second] [DEBUG] [06/14/2012 08:19:03.224] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=2996, cap=2996)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.225] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2996, cap=2996)
[third] [DEBUG] [06/14/2012 08:19:03.221] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:03.222] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.222] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:03.222] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[third] [DEBUG] [06/14/2012 08:19:03.222] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:03.222] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:03.222] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=852, cap=852)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.223] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.223] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=852, cap=852)
[third] [DEBUG] [06/14/2012 08:19:03.223] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.223] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.224] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5e725967),CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.224] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.224] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:03.225] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:03.228] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.228] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.277] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:03.277] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:03.278] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.278] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:03.278] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.278] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to members not in seen table [Set(akka://ConvergenceSpec@second)].
[first] [DEBUG] [06/14/2012 08:19:03.278] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:03.278] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.280] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.280] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1029f93b),CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.280] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.280] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[third] [DEBUG] [06/14/2012 08:19:03.280] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=788, cap=788)
[third] [DEBUG] [06/14/2012 08:19:03.280] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.281] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[third] [DEBUG] [06/14/2012 08:19:03.281] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=788, cap=788)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.281] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=788, cap=788)
[third] [DEBUG] [06/14/2012 08:19:03.283] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:03.284] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:03.317] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:03.317] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:03.318] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.318] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:03.318] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.319] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.318] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:03.319] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.319] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:03.319] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.319] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.320] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.320] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.320] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@254ba9a2),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.320] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.320] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.321] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:03.321] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:03.321] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.321] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@32fd4662),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.321] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.318] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6e75d758),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.318] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.319] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.319] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@41d144a4),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.319] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.321] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.319] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.319] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.320] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.321] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.321] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:03.322] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.322] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.320] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:03.320] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.321] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.321] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.322] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:03.322] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.416] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:03.416] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:03.416] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:03.416] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.416] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.417] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:03.417] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.418] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:03.418] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.418] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.419] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4f549ceb),CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.419] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:03.419] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.419] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:03.419] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:03.419] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.419] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3056, cap=3056)
[third] [DEBUG] [06/14/2012 08:19:03.420] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=3056, cap=3056)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.420] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3056, cap=3056)
[third] [DEBUG] [06/14/2012 08:19:03.420] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.421] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.421] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:03.422] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.422] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5b62f030),CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.422] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.422] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:03.422] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=948, cap=948)
[first] [DEBUG] [06/14/2012 08:19:03.422] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.423] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:03.423] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=948, cap=948)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.423] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.423] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:03.423] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=948, cap=948)
[third] [DEBUG] [06/14/2012 08:19:03.424] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:03.424] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.425] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@679b2faf),CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.425] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.425] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2996, cap=2996)
[second] [DEBUG] [06/14/2012 08:19:03.425] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=2996, cap=2996)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.425] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2996, cap=2996)
[second] [DEBUG] [06/14/2012 08:19:03.429] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.429] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.431] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:03.432] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.475] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:03.476] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:03.476] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:03.476] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.476] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.476] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:03.476] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)]
[first] [INFO] [06/14/2012 08:19:03.477] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Leader is moving node [akka://ConvergenceSpec@first] from JOINING to UP
[first] [INFO] [06/14/2012 08:19:03.477] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Leader is moving node [akka://ConvergenceSpec@third] from JOINING to UP
[first] [DEBUG] [06/14/2012 08:19:03.477] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.478] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.479] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1a83e35b),CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.479] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.479] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)
[first] [DEBUG] [06/14/2012 08:19:03.479] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[third] [DEBUG] [06/14/2012 08:19:03.480] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.480] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)
[third] [DEBUG] [06/14/2012 08:19:03.483] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:03.483] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:03.574] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:03.574] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.574] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.575] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:03.575] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.575] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@48c378f6),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.575] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.575] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.575] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.575] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@50a2d304),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.576] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.576] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.576] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.576] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:03.576] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.576] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:03.577] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.577] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:03.577] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:03.577] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:03.580] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:03.613] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:03.614] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:03.614] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:03.614] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.614] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.614] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:03.617] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[third] [DEBUG] [06/14/2012 08:19:03.616] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:03.617] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2032, cap=2032)
[first] [DEBUG] [06/14/2012 08:19:03.617] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.617] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[first] [DEBUG] [06/14/2012 08:19:03.617] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=2032, cap=2032)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.617] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2032, cap=2032)
[second] [DEBUG] [06/14/2012 08:19:03.615] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:03.616] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.616] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@29e26602),CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.616] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.616] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:03.616] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:03.617] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:03.617] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:03.617] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:03.619] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[third] [DEBUG] [06/14/2012 08:19:03.618] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.619] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.620] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.619] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4a0ce796),CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.619] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.620] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[first] [DEBUG] [06/14/2012 08:19:03.621] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:03.622] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:03.624] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.624] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.673] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:03.673] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:03.673] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:03.673] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.673] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.674] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[first] [DEBUG] [06/14/2012 08:19:03.674] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:03.674] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.675] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2866, cap=2866, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.676] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@73a01e8a),CompositeChannelBuffer(ridx=0, widx=2866, cap=2866, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.676] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2866, cap=2866, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.676] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:03.676] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=818, cap=818)
[second] [DEBUG] [06/14/2012 08:19:03.676] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.677] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:03.677] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=818, cap=818)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.677] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=818, cap=818)
[second] [DEBUG] [06/14/2012 08:19:03.680] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.680] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:03.680] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:03.713] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:03.713] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:03.713] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:03.714] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.714] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.715] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.714] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:03.715] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.716] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.716] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:03.716] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.716] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.716] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.717] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@74b70648),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.714] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@246ce26e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.714] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.714] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:03.715] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.715] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@228ca8ac),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.715] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.718] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:03.718] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.718] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:03.717] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.717] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.717] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:03.717] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:03.717] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.715] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.715] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.716] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:03.716] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:03.718] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2743724e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.719] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.719] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:03.718] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.717] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.717] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.717] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:03.718] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:03.718] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.781] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:03.812] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:03.812] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:03.813] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:03.813] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.814] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.814] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:03.814] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[third] [DEBUG] [06/14/2012 08:19:03.815] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[third] [DEBUG] [06/14/2012 08:19:03.815] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:03.816] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:03.816] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:03.816] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.821] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[third] [DEBUG] [06/14/2012 08:19:03.816] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.821] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=878, cap=878)
[third] [DEBUG] [06/14/2012 08:19:03.817] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.819] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.821] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.822] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:03.822] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@299481b2),CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.822] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:03.822] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.820] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.820] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@56d99277),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.820] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.826] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[second] [DEBUG] [06/14/2012 08:19:03.826] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.826] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[second] [DEBUG] [06/14/2012 08:19:03.830] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.830] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:03.823] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:03.825] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[third] [DEBUG] [06/14/2012 08:19:03.825] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1bae939f),CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.826] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.821] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.821] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:03.821] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=878, cap=878)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.822] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=878, cap=878)
[first] [DEBUG] [06/14/2012 08:19:03.822] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[first] [DEBUG] [06/14/2012 08:19:03.823] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.823] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[first] [DEBUG] [06/14/2012 08:19:03.827] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:03.828] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:03.828] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:03.828] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.871] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:03.871] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:03.871] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:03.871] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.871] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:03.871] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:03.874] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[first] [DEBUG] [06/14/2012 08:19:03.871] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:03.874] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.874] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[first] [DEBUG] [06/14/2012 08:19:03.872] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.873] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.873] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@54e0d16b),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.873] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.877] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.877] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:03.881] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:03.914] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[first] [DEBUG] [06/14/2012 08:19:03.970] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:03.970] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.970] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.971] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:03.971] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:03.971] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@73d4f355),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.971] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.971] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:03.971] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@12e4d0cd),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:03.971] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:03.971] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:03.971] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:03.972] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:03.972] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:03.972] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:03.972] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:03.972] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:03.973] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:03.973] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:03.973] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:03.981] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:04.010] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:04.010] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:04.010] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:04.011] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.011] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.011] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:04.013] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:04.012] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.014] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.014] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@71107304),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.014] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.015] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:04.018] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[second] [DEBUG] [06/14/2012 08:19:04.018] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.018] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[third] [DEBUG] [06/14/2012 08:19:04.013] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:04.013] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:04.014] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:04.014] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:04.014] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:04.014] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[third] [DEBUG] [06/14/2012 08:19:04.015] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.015] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[third] [DEBUG] [06/14/2012 08:19:04.015] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:04.017] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.017] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4e9722c9),CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.017] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.020] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.020] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:04.022] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.022] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.069] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:04.070] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:04.070] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:04.070] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.070] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.070] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[first] [DEBUG] [06/14/2012 08:19:04.070] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:04.071] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:04.073] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.073] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1ec278b5),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.073] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.073] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[second] [DEBUG] [06/14/2012 08:19:04.074] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.074] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[second] [DEBUG] [06/14/2012 08:19:04.078] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.078] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:04.082] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[third] [DEBUG] [06/14/2012 08:19:04.086] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)]
[third] [DEBUG] [06/14/2012 08:19:04.087] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-second-third-joined)
[third] [DEBUG] [06/14/2012 08:19:04.088] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-second-third-joined)),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$c]
[third] [DEBUG] [06/14/2012 08:19:04.088] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711] written 33
[first] [DEBUG] [06/14/2012 08:19:04.088] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61376: EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:04.088] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(EnterBarrier(first-second-third-joined),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.089] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:04.089] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-second-third-joined),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]
[first] [DEBUG] [06/14/2012 08:19:04.089] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])))
[second] [DEBUG] [06/14/2012 08:19:04.109] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:04.109] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.109] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.110] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:04.110] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:04.111] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:04.110] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:04.111] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:04.111] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.111] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:04.111] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:04.111] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.112] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:04.112] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@565539d8),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.112] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.110] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@37c2ec05),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.110] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.110] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.111] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4b7d03c5),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.111] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.113] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:04.113] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.113] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:04.114] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.114] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.115] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[third] [DEBUG] [06/14/2012 08:19:04.112] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.112] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.112] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:04.112] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.112] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7b4189d0),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.113] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.111] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.111] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:04.112] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.112] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:04.112] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:04.112] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.113] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:04.113] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.113] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.182] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:04.208] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:04.208] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:04.209] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:04.209] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.209] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.209] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:04.211] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:04.210] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[third] [DEBUG] [06/14/2012 08:19:04.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:04.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:04.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:04.212] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:04.212] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:04.213] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.213] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.213] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:04.213] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.214] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4e1d2b8e),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.214] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[second] [DEBUG] [06/14/2012 08:19:04.214] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.214] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.214] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[second] [DEBUG] [06/14/2012 08:19:04.215] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.215] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[third] [DEBUG] [06/14/2012 08:19:04.216] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.216] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7593c366),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.216] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.217] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[second] [DEBUG] [06/14/2012 08:19:04.217] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.219] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.217] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[first] [DEBUG] [06/14/2012 08:19:04.219] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:04.217] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6d854b38),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.217] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.217] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[second] [DEBUG] [06/14/2012 08:19:04.218] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.221] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.221] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:04.218] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.218] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[third] [DEBUG] [06/14/2012 08:19:04.222] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.223] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:04.267] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:04.267] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:04.268] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.268] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:04.268] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.268] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:04.270] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[second] [DEBUG] [06/14/2012 08:19:04.270] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.270] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[second] [DEBUG] [06/14/2012 08:19:04.273] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.274] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:04.268] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[first] [DEBUG] [06/14/2012 08:19:04.268] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:04.269] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.270] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@44285d14),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.270] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.283] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:04.316] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[first] [DEBUG] [06/14/2012 08:19:04.366] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:04.366] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:04.366] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.366] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:04.366] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:04.367] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@35f6ef01),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.367] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.367] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.367] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@218c6982),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.367] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.367] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:04.367] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.367] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:04.367] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:04.367] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.368] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:04.368] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.368] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.368] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:04.369] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:04.383] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:04.406] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:04.406] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:04.406] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:04.406] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.406] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.406] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:04.408] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:04.407] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.408] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@29f2588),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:04.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:04.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:04.409] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:04.409] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:04.409] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[third] [DEBUG] [06/14/2012 08:19:04.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[third] [DEBUG] [06/14/2012 08:19:04.410] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:04.411] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.411] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:04.411] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@49b29f80),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.411] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:04.411] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.411] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[first] [DEBUG] [06/14/2012 08:19:04.412] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.412] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[third] [DEBUG] [06/14/2012 08:19:04.412] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:04.413] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.413] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52256, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.413] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@37c2ec05),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.413] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.413] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:04.414] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[second] [DEBUG] [06/14/2012 08:19:04.414] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.414] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[first] [DEBUG] [06/14/2012 08:19:04.415] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.415] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.416] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:04.418] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.418] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.465] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:04.465] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:04.466] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:04.466] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.466] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.466] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:04.466] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)]
[first] [DEBUG] [06/14/2012 08:19:04.466] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:04.468] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.468] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4e6d670a),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.468] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.469] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[third] [DEBUG] [06/14/2012 08:19:04.469] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1f7d2f0e),BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.469] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[third] [DEBUG] [06/14/2012 08:19:04.472] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.472] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:04.484] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)]
[first] [DEBUG] [06/14/2012 08:19:04.485] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:04.485] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-second-third-joined)),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$b]
[first] [DEBUG] [06/14/2012 08:19:04.486] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 33
[first] [DEBUG] [06/14/2012 08:19:04.486] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:04.486] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(EnterBarrier(first-second-third-joined),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.486] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:04.486] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-second-third-joined),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]
[first] [DEBUG] [06/14/2012 08:19:04.487] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])))
[second] [DEBUG] [06/14/2012 08:19:04.505] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:04.505] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.506] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.506] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:04.506] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.506] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7c514f7f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.507] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:04.506] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:04.507] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1d4fcd9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.507] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:04.508] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:04.508] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:04.508] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.508] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:04.508] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e19fc25, /127.0.0.1:61379 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.508] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:04.509] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:04.509] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1b275a34,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5e1558dc),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.509] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.509] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x66f9104a, /127.0.0.1:61380 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:04.509] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1abbbd0e,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7c82a13a),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:04.509] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.506] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.506] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.507] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6912d7ae),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.507] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.509] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:04.510] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@103c37f4),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.510] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:04.511] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.511] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.507] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.507] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:04.508] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.508] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:04.509] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:04.509] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@7307f742),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.509] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:04.510] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.510] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.516] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)]
[second] [DEBUG] [06/14/2012 08:19:04.518] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-second-third-joined)
[second] [DEBUG] [06/14/2012 08:19:04.518] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-second-third-joined)),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$c]
[second] [DEBUG] [06/14/2012 08:19:04.518] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 33
[first] [DEBUG] [06/14/2012 08:19:04.519] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:04.519] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(EnterBarrier(first-second-third-joined),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.520] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:04.520] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-second-third-joined),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]
[first] [DEBUG] [06/14/2012 08:19:04.521] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])))
[first] [DEBUG] [06/14/2012 08:19:04.521] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] canceling timer 'Timeout'
[first] [DEBUG] [06/14/2012 08:19:04.521] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(ToClient(BarrierResult(first-second-third-joined,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:04.521] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(BarrierResult(first-second-third-joined,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:04.521] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] transition Waiting -> Idle
[first] [DEBUG] [06/14/2012 08:19:04.521] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(BarrierResult(first-second-third-joined,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:04.521] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(BarrierResult(first-second-third-joined,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[third] [DEBUG] [06/14/2012 08:19:04.521] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-second-third-joined,true)
[third] [DEBUG] [06/14/2012 08:19:04.522] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-second-third-joined,true),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),Some((first-second-third-joined,Actor[akka://ConvergenceSpec/temp/$c])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.522] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-second-third-joined
[fourth] [DEBUG] [06/14/2012 08:19:04.522] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-second-third-joined,true)
[second] [DEBUG] [06/14/2012 08:19:04.522] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-second-third-joined,true)
[second] [DEBUG] [06/14/2012 08:19:04.522] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-second-third-joined,true),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((first-second-third-joined,Actor[akka://ConvergenceSpec/temp/$c])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:04.522] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-second-third-joined,true),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Some((first-second-third-joined,Actor[akka://ConvergenceSpec/temp/$b])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:04.523] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-second-third-joined
[second] [DEBUG] [06/14/2012 08:19:04.523] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-second-third-joined
[first] [DEBUG] [06/14/2012 08:19:04.522] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: BarrierResult(first-second-third-joined,true)
[first] [DEBUG] [06/14/2012 08:19:04.522] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-second-third-joined,true),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((first-second-third-joined,Actor[akka://ConvergenceSpec/temp/$b])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.522] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-second-third-joined
[first] [DEBUG] [06/14/2012 08:19:04.523] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-1)
[first] [DEBUG] [06/14/2012 08:19:04.523] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-1)),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$c]
[fourth] [DEBUG] [06/14/2012 08:19:04.523] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-1)
[fourth] [DEBUG] [06/14/2012 08:19:04.524] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-1)),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$c]
[third] [DEBUG] [06/14/2012 08:19:04.523] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-1)
[third] [DEBUG] [06/14/2012 08:19:04.523] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-1)),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$d]
[third] [DEBUG] [06/14/2012 08:19:04.524] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:04.524] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:04.524] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61376: EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:04.525] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: EnterBarrier(after-1)
[fourth] [DEBUG] [06/14/2012 08:19:04.524] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:04.525] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:04.525] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(EnterBarrier(after-1),None) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.524] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-1)
[second] [DEBUG] [06/14/2012 08:19:04.524] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-1)),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$d]
[second] [DEBUG] [06/14/2012 08:19:04.525] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:04.525] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(EnterBarrier(after-1),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.525] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:04.525] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(EnterBarrier(after-1),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.525] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:04.526] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(EnterBarrier(after-1),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.526] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:04.526] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:04.526] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:04.526] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-1),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]
[first] [DEBUG] [06/14/2012 08:19:04.526] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] setting timer 'Timeout'/30000 milliseconds: StateTimeout
[first] [DEBUG] [06/14/2012 08:19:04.526] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] transition Idle -> Waiting
[first] [DEBUG] [06/14/2012 08:19:04.526] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-1),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]
[first] [DEBUG] [06/14/2012 08:19:04.527] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3])))
[first] [DEBUG] [06/14/2012 08:19:04.527] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-1),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]
[first] [DEBUG] [06/14/2012 08:19:04.528] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3])))
[first] [DEBUG] [06/14/2012 08:19:04.528] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-1),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]
[first] [DEBUG] [06/14/2012 08:19:04.529] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3])))
[first] [DEBUG] [06/14/2012 08:19:04.529] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] canceling timer 'Timeout'
[first] [DEBUG] [06/14/2012 08:19:04.529] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(BarrierResult(after-1,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:04.529] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(BarrierResult(after-1,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:04.529] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(BarrierResult(after-1,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:04.530] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] transition Waiting -> Idle
[first] [DEBUG] [06/14/2012 08:19:04.530] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(ToClient(BarrierResult(after-1,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:04.530] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: BarrierResult(after-1,true)
[second] [DEBUG] [06/14/2012 08:19:04.530] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-1,true)
[second] [DEBUG] [06/14/2012 08:19:04.530] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-1,true),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((after-1,Actor[akka://ConvergenceSpec/temp/$d])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:04.530] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-1,true)
[second] [DEBUG] [06/14/2012 08:19:04.530] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-1
[fourth] [DEBUG] [06/14/2012 08:19:04.531] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-1,true),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Some((after-1,Actor[akka://ConvergenceSpec/temp/$c])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:04.531] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-1
[third] [DEBUG] [06/14/2012 08:19:04.531] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-1,true)
[third] [DEBUG] [06/14/2012 08:19:04.531] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-1,true),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),Some((after-1,Actor[akka://ConvergenceSpec/temp/$d])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.530] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-1,true),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((after-1,Actor[akka://ConvergenceSpec/temp/$c])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.531] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-1
[third] [DEBUG] [06/14/2012 08:19:04.532] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-1
[second] - must reach initial convergence
[first] - must reach initial convergence
[first] [DEBUG] [06/14/2012 08:19:04.534] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(third))),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$d]
[first] [DEBUG] [06/14/2012 08:19:04.534] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:04.534] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:04.535] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: GetAddress(RoleName(third))
[fourth] - must reach initial convergence
[second] [DEBUG] [06/14/2012 08:19:04.534] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(third))),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$e]
[second] [DEBUG] [06/14/2012 08:19:04.534] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 13
[third] - must reach initial convergence
[first] [DEBUG] [06/14/2012 08:19:04.535] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(GetAddress(RoleName(third)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.535] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(GetAddress(RoleName(third)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.535] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:04.535] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:04.535] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(AddressReply(RoleName(third),akka://ConvergenceSpec@third)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:04.536] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(AddressReply(RoleName(third),akka://ConvergenceSpec@third)),None) from Actor[akka://ConvergenceSpec/user/controller]
[third] [DEBUG] [06/14/2012 08:19:04.536] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(third))),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$e]
[first] [DEBUG] [06/14/2012 08:19:04.536] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(third),akka://ConvergenceSpec@third)
[second] [DEBUG] [06/14/2012 08:19:04.536] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(third),akka://ConvergenceSpec@third)
[first] [DEBUG] [06/14/2012 08:19:04.536] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(third),akka://ConvergenceSpec@third),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((third,Actor[akka://ConvergenceSpec/temp/$d])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.537] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61376: GetAddress(RoleName(third))
[third] [DEBUG] [06/14/2012 08:19:04.536] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711] written 13
[fourth] [DEBUG] [06/14/2012 08:19:04.536] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(third))),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$d]
[first] [DEBUG] [06/14/2012 08:19:04.537] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(GetAddress(RoleName(third)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.537] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(third),akka://ConvergenceSpec@third),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((third,Actor[akka://ConvergenceSpec/temp/$e])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.537] [main] [ActorSystem(ConvergenceSpec)] entering barriers (before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:04.537] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:04.537] [main] [ActorSystem(ConvergenceSpec)] entering barriers (before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:04.537] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(ToClient(AddressReply(RoleName(third),akka://ConvergenceSpec@third)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:04.537] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(before-shutdown)),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$e]
[second] [DEBUG] [06/14/2012 08:19:04.537] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(before-shutdown)),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$f]
[first] [DEBUG] [06/14/2012 08:19:04.537] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 23
[third] [DEBUG] [06/14/2012 08:19:04.537] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(third),akka://ConvergenceSpec@third)
[second] [DEBUG] [06/14/2012 08:19:04.538] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 23
[first] [DEBUG] [06/14/2012 08:19:04.538] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:04.538] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: EnterBarrier(before-shutdown)
[third] [DEBUG] [06/14/2012 08:19:04.538] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(third),akka://ConvergenceSpec@third),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),Some((third,Actor[akka://ConvergenceSpec/temp/$e])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.538] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(EnterBarrier(before-shutdown),None) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.538] [main] [ActorSystem(ConvergenceSpec)] entering barriers (before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:04.538] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:04.538] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(EnterBarrier(before-shutdown),None) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.538] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(before-shutdown)),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$f]
[first] [DEBUG] [06/14/2012 08:19:04.539] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(before-shutdown)
[fourth] [DEBUG] [06/14/2012 08:19:04.538] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:04.539] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(before-shutdown),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]
[third] [DEBUG] [06/14/2012 08:19:04.539] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711] written 23
[first] [DEBUG] [06/14/2012 08:19:04.539] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:04.539] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] setting timer 'Timeout'/30000 milliseconds: StateTimeout
[first] [DEBUG] [06/14/2012 08:19:04.539] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(GetAddress(RoleName(third)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.539] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] transition Idle -> Waiting
[first] [DEBUG] [06/14/2012 08:19:04.539] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:04.539] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61376: EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:04.540] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(AddressReply(RoleName(third),akka://ConvergenceSpec@third)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:04.540] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(EnterBarrier(before-shutdown),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.540] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:04.540] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(before-shutdown),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]
[first] [DEBUG] [06/14/2012 08:19:04.540] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1])))
[fourth] [DEBUG] [06/14/2012 08:19:04.540] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(third),akka://ConvergenceSpec@third)
[first] [DEBUG] [06/14/2012 08:19:04.540] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(before-shutdown),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]
[fourth] [DEBUG] [06/14/2012 08:19:04.540] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(third),akka://ConvergenceSpec@third),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Some((third,Actor[akka://ConvergenceSpec/temp/$d])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.541] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1])))
[fourth] [DEBUG] [06/14/2012 08:19:04.541] [main] [ActorSystem(ConvergenceSpec)] entering barriers (before-shutdown)
[fourth] [DEBUG] [06/14/2012 08:19:04.541] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(before-shutdown)),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$e]
[fourth] [DEBUG] [06/14/2012 08:19:04.541] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 23
[first] [DEBUG] [06/14/2012 08:19:04.541] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:04.541] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(EnterBarrier(before-shutdown),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.541] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:04.542] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(before-shutdown),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]
[first] [DEBUG] [06/14/2012 08:19:04.542] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1])))
[first] [DEBUG] [06/14/2012 08:19:04.542] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] canceling timer 'Timeout'
[first] [DEBUG] [06/14/2012 08:19:04.542] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] transition Waiting -> Idle
[first] [DEBUG] [06/14/2012 08:19:04.542] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(ToClient(BarrierResult(before-shutdown,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:04.543] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(BarrierResult(before-shutdown,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[fourth] [DEBUG] [06/14/2012 08:19:04.543] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(before-shutdown,true)
[second] [DEBUG] [06/14/2012 08:19:04.543] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(before-shutdown,true)
[first] [DEBUG] [06/14/2012 08:19:04.543] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(BarrierResult(before-shutdown,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:04.543] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(BarrierResult(before-shutdown,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:04.543] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: BarrierResult(before-shutdown,true)
[second] [DEBUG] [06/14/2012 08:19:04.543] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(before-shutdown,true),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((before-shutdown,Actor[akka://ConvergenceSpec/temp/$f])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.543] [main] [ActorSystem(ConvergenceSpec)] passed barrier before-shutdown
[fourth] [DEBUG] [06/14/2012 08:19:04.543] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(before-shutdown,true),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Some((before-shutdown,Actor[akka://ConvergenceSpec/temp/$e])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.543] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(before-shutdown,true)
[third] [DEBUG] [06/14/2012 08:19:04.543] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(before-shutdown,true),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),Some((before-shutdown,Actor[akka://ConvergenceSpec/temp/$f])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.543] [main] [ActorSystem(ConvergenceSpec)] passed barrier before-shutdown
[first] [DEBUG] [06/14/2012 08:19:04.543] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(before-shutdown,true),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((before-shutdown,Actor[akka://ConvergenceSpec/temp/$e])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.543] [main] [ActorSystem(ConvergenceSpec)] passed barrier before-shutdown
[fourth] [DEBUG] [06/14/2012 08:19:04.544] [main] [ActorSystem(ConvergenceSpec)] passed barrier before-shutdown
[third] [DEBUG] [06/14/2012 08:19:04.544] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-2)
[first] [DEBUG] [06/14/2012 08:19:04.545] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message Terminate(RoleName(third),0)
[fourth] [DEBUG] [06/14/2012 08:19:04.545] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-2)
[fourth] [DEBUG] [06/14/2012 08:19:04.545] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-2)),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$f]
[third] [DEBUG] [06/14/2012 08:19:04.545] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-2)),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$g]
[third] [DEBUG] [06/14/2012 08:19:04.545] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:04.545] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:04.546] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61376: EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:04.546] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: GetAddress(RoleName(first))
[fourth] [DEBUG] [06/14/2012 08:19:04.545] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 15
[second] [DEBUG] [06/14/2012 08:19:04.545] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$g]
[first] [DEBUG] [06/14/2012 08:19:04.546] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(EnterBarrier(after-2),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.546] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(EnterBarrier(after-2),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.546] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.546] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:04.546] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:04.546] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(ToClient(TerminateMsg(0)),None) from Actor[akka://ConvergenceSpec/temp/$f]
[first] [DEBUG] [06/14/2012 08:19:04.547] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:04.547] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] processing Event(RemoveClient(RoleName(third)),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:04.547] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:04.547] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$g]
[first] [DEBUG] [06/14/2012 08:19:04.547] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:04.547] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:04.547] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.548] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:04.548] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-2),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3]
[first] [DEBUG] [06/14/2012 08:19:04.548] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:04.548] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(ToClient(BarrierResult(after-2,false)),Some(Actor[akka://ConvergenceSpec/temp/$f])) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:04.548] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-2),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]
[first] [DEBUG] [06/14/2012 08:19:04.548] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[first] [DEBUG] [06/14/2012 08:19:04.549] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] setting timer 'Timeout'/30000 milliseconds: StateTimeout
[first] [DEBUG] [06/14/2012 08:19:04.549] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] transition Idle -> Waiting
[first] [DEBUG] [06/14/2012 08:19:04.549] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$g])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.549] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:04.549] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(GetAddress(RoleName(second)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.549] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:04.549] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(second))),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$h]
[first] [DEBUG] [06/14/2012 08:19:04.549] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(AddressReply(RoleName(second),akka://ConvergenceSpec@second)),None) from Actor[akka://ConvergenceSpec/user/controller]
[third] [DEBUG] [06/14/2012 08:19:04.547] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: TerminateMsg(0)
[third] [DEBUG] [06/14/2012 08:19:04.548] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(TerminateMsg(0),Data(Some([id: 0x0fc8fb4b, /127.0.0.1:61376 => localhost/127.0.0.1:4711]),Some((after-2,Actor[akka://ConvergenceSpec/temp/$g])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:04.548] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-2,false)
[first] [DEBUG] [06/14/2012 08:19:04.549] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 14
[first] [DEBUG] [06/14/2012 08:19:04.550] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:04.550] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(GetAddress(RoleName(second)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.550] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:04.550] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(AddressReply(RoleName(second),akka://ConvergenceSpec@second)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:04.550] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(second),akka://ConvergenceSpec@second)
[first] [DEBUG] [06/14/2012 08:19:04.550] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(second),akka://ConvergenceSpec@second),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((second,Actor[akka://ConvergenceSpec/temp/$h])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.545] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 13
[second] [DEBUG] [06/14/2012 08:19:04.547] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[second] [DEBUG] [06/14/2012 08:19:04.547] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$g])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.548] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(second))),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$h]
[second] [DEBUG] [06/14/2012 08:19:04.548] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 14
[second] [DEBUG] [06/14/2012 08:19:04.549] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(second),akka://ConvergenceSpec@second)
[second] [DEBUG] [06/14/2012 08:19:04.550] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(second),akka://ConvergenceSpec@second),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((second,Actor[akka://ConvergenceSpec/temp/$h])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.604] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:04.604] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:04.604] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:04.604] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.604] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.604] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:04.605] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.606] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.606] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6a095b8a),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.607] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.607] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[first] [DEBUG] [06/14/2012 08:19:04.607] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.607] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[first] [DEBUG] [06/14/2012 08:19:04.610] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.610] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:04.664] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:04.664] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:04.664] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:04.664] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.664] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.664] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:04.664] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)]
[first] [INFO] [06/14/2012 08:19:04.665] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Marking node(s) as UNREACHABLE [Member(address = akka://ConvergenceSpec@third, status = Up)]
[first] [DEBUG] [06/14/2012 08:19:04.665] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:04.667] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b1402c4, localhost/127.0.0.1:61379 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.667] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@344e17f9,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2a738a71),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.667] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61379-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.763] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:04.763] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:04.763] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.764] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1867ec28),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.764] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.764] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:04.764] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.764] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:04.765] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.765] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:04.802] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:04.802] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:04.802] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:04.803] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.803] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.803] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:04.808] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[second] [DEBUG] [06/14/2012 08:19:04.804] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.806] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.806] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.807] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:04.807] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6f184a4f),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.807] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.808] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.809] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x037c398e, localhost/127.0.0.1:61380 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.809] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5aacc14,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@feeb372),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.810] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61380-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.808] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.808] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[first] [DEBUG] [06/14/2012 08:19:04.811] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.812] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:04.861] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:04.861] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:04.861] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:04.861] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:04.861] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:04.862] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:04.862] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:04.864] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.865] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@40944dbd),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.866] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.866] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [DEBUG] [06/14/2012 08:19:04.866] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:04.866] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [DEBUG] [06/14/2012 08:19:04.870] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.871] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:04.877] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x037c398e, localhost/127.0.0.1:61380 :> /127.0.0.1:61367]
[first] [DEBUG] [06/14/2012 08:19:04.877] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] disconnect from /127.0.0.1:61376
[first] [DEBUG] [06/14/2012 08:19:04.877] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x1b1402c4, localhost/127.0.0.1:61379 :> /127.0.0.1:61364]
[first] [DEBUG] [06/14/2012 08:19:04.878] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61376-server3] processing Event(ClientDisconnected,Some(Actor[akka://ConvergenceSpec/temp/$f])) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:04.879] [ConvergenceSpec-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [PassiveRemoteClient@akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.880] [ConvergenceSpec-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [PassiveRemoteClient@akka://ConvergenceSpec@third]
[second] [INFO] [06/14/2012 08:19:04.880] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:04.880] [ConvergenceSpec-6] [RemoteClient(akka://ConvergenceSpec)] [PassiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [INFO] [06/14/2012 08:19:04.881] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:04.881] [ConvergenceSpec-3] [RemoteClient(akka://ConvergenceSpec)] [PassiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[second] [DEBUG] [06/14/2012 08:19:04.882] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteServerClientClosed@akka://ConvergenceSpec@second: Client[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:04.882] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteServerClientClosed@akka://ConvergenceSpec@first: Client[akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:04.901] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:04.901] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:04.902] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:04.902] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4a9b601d),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.903] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:04.902] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:04.903] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:04.903] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:04.904] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.904] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:04.960] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[second] [DEBUG] [06/14/2012 08:19:04.961] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-2)
[second] [DEBUG] [06/14/2012 08:19:04.961] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-2)),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$i]
[second] [DEBUG] [06/14/2012 08:19:04.962] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:04.962] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:04.962] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(EnterBarrier(after-2),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:04.962] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:04.962] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-2),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-2,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]
[first] [DEBUG] [06/14/2012 08:19:04.963] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-2,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])))
[second] [DEBUG] [06/14/2012 08:19:05.000] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:05.000] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:05.000] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.000] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.000] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:05.001] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.004] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [DEBUG] [06/14/2012 08:19:05.003] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.003] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@203c9616),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.003] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.004] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.004] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[first] [DEBUG] [06/14/2012 08:19:05.007] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.007] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:05.059] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:05.059] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:05.059] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:05.059] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:05.059] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:05.060] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:05.060] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:05.060] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.062] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-2)
[first] [DEBUG] [06/14/2012 08:19:05.062] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-2)),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$i]
[first] [DEBUG] [06/14/2012 08:19:05.062] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.063] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:05.063] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:05.063] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@152e7cb6),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.063] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.063] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(EnterBarrier(after-2),None) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.063] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[first] [DEBUG] [06/14/2012 08:19:05.063] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:05.063] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-2),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-2,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]
[second] [DEBUG] [06/14/2012 08:19:05.063] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.064] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-2,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])))
[first] [DEBUG] [06/14/2012 08:19:05.064] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] canceling timer 'Timeout'
[second] [DEBUG] [06/14/2012 08:19:05.063] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[first] [DEBUG] [06/14/2012 08:19:05.064] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(BarrierResult(after-2,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:05.064] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] transition Waiting -> Idle
[first] [DEBUG] [06/14/2012 08:19:05.064] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(BarrierResult(after-2,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:05.064] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(BarrierResult(after-2,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:05.064] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: BarrierResult(after-2,true)
[second] [DEBUG] [06/14/2012 08:19:05.064] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-2,true)
[fourth] [DEBUG] [06/14/2012 08:19:05.064] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-2,true)
[second] [DEBUG] [06/14/2012 08:19:05.065] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-2,true),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((after-2,Actor[akka://ConvergenceSpec/temp/$i])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.065] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-2
[first] [DEBUG] [06/14/2012 08:19:05.065] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-2,true),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((after-2,Actor[akka://ConvergenceSpec/temp/$i])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.065] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-2
[second] - must not reach convergence while any nodes are unreachable
[fourth] [DEBUG] [06/14/2012 08:19:05.065] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-2,true),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Some((after-2,Actor[akka://ConvergenceSpec/temp/$f])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:05.065] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-2
[fourth] - must not reach convergence while any nodes are unreachable
[first] - must not reach convergence while any nodes are unreachable
[first] [DEBUG] [06/14/2012 08:19:05.066] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$j]
[first] [DEBUG] [06/14/2012 08:19:05.067] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 13
[second] [DEBUG] [06/14/2012 08:19:05.067] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$j]
[first] [DEBUG] [06/14/2012 08:19:05.067] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:05.067] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.067] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: GetAddress(RoleName(first))
[second] [DEBUG] [06/14/2012 08:19:05.067] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:05.067] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:05.068] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.068] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:05.068] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:05.068] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:05.068] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[second] [DEBUG] [06/14/2012 08:19:05.068] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.068] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[second] [DEBUG] [06/14/2012 08:19:05.069] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:05.069] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$j])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.069] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$j])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.069] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(second))),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$k]
[second] [DEBUG] [06/14/2012 08:19:05.069] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(second))),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$k]
[first] [DEBUG] [06/14/2012 08:19:05.069] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 14
[second] [DEBUG] [06/14/2012 08:19:05.069] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 14
[first] [DEBUG] [06/14/2012 08:19:05.070] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:05.070] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:05.070] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(GetAddress(RoleName(second)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.070] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(GetAddress(RoleName(second)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.070] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:05.070] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:05.070] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(AddressReply(RoleName(second),akka://ConvergenceSpec@second)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:05.070] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(AddressReply(RoleName(second),akka://ConvergenceSpec@second)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:05.071] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(second),akka://ConvergenceSpec@second)
[second] [DEBUG] [06/14/2012 08:19:05.071] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(second),akka://ConvergenceSpec@second)
[first] [DEBUG] [06/14/2012 08:19:05.071] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(second),akka://ConvergenceSpec@second),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((second,Actor[akka://ConvergenceSpec/temp/$k])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.071] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(second),akka://ConvergenceSpec@second),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((second,Actor[akka://ConvergenceSpec/temp/$k])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.071] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(fourth))),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$l]
[second] [DEBUG] [06/14/2012 08:19:05.072] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(fourth))),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$l]
[first] [DEBUG] [06/14/2012 08:19:05.072] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 14
[first] [DEBUG] [06/14/2012 08:19:05.072] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: GetAddress(RoleName(fourth))
[second] [DEBUG] [06/14/2012 08:19:05.072] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 14
[first] [DEBUG] [06/14/2012 08:19:05.072] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: GetAddress(RoleName(fourth))
[first] [DEBUG] [06/14/2012 08:19:05.072] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(GetAddress(RoleName(fourth)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.073] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(GetAddress(RoleName(fourth)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.073] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(fourth))
[first] [DEBUG] [06/14/2012 08:19:05.073] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(fourth))
[first] [DEBUG] [06/14/2012 08:19:05.073] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:05.073] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:05.073] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)
[second] [DEBUG] [06/14/2012 08:19:05.073] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)
[first] [DEBUG] [06/14/2012 08:19:05.074] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((fourth,Actor[akka://ConvergenceSpec/temp/$l])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.074] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((fourth,Actor[akka://ConvergenceSpec/temp/$l])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.075] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode1(akka://ConvergenceSpec)] assertNotMovedUp#1
[second] [DEBUG] [06/14/2012 08:19:05.075] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode2(akka://ConvergenceSpec)] assertNotMovedUp#1
[fourth] [INFO] [06/14/2012 08:19:05.154] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - is starting up...
[first] [DEBUG] [06/14/2012 08:19:05.158] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:05.159] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.159] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.159] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4cd297c0),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.159] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.159] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:05.160] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.160] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:05.160] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.161] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [INFO] [06/14/2012 08:19:05.175] [main] [Node(akka://ConvergenceSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[fourth] [INFO] [06/14/2012 08:19:05.190] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - registering cluster JMX MBean [akka:type=Cluster]
[fourth] [INFO] [06/14/2012 08:19:05.190] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - has started up successfully
[fourth] [DEBUG] [06/14/2012 08:19:05.191] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$g]
[fourth] [DEBUG] [06/14/2012 08:19:05.191] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:05.191] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:05.191] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.198] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:05.198] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:05.198] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.199] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.199] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:05.192] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[first] [DEBUG] [06/14/2012 08:19:05.191] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:05.192] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[fourth] [DEBUG] [06/14/2012 08:19:05.192] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$g])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [INFO] [06/14/2012 08:19:05.193] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Trying to send JOIN to [akka://ConvergenceSpec@first] through connection [Actor[akka://ConvergenceSpec@first/system/cluster/commands]]
[fourth] [DEBUG] [06/14/2012 08:19:05.197] [main] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@first|localhost/127.0.0.1]
[second] [DEBUG] [06/14/2012 08:19:05.199] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:05.201] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.201] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7331f919),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.201] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.201] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:05.202] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [DEBUG] [06/14/2012 08:19:05.202] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:05.202] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:05.202] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.202] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [ERROR] [06/14/2012 08:19:05.208] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:05.208] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [DEBUG] [06/14/2012 08:19:05.209] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:05.210] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x55d91364] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.210] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2) in direction Send when socket [id: 0x55d91364] already closed, dropping
[second] [DEBUG] [06/14/2012 08:19:05.211] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [INFO] [06/14/2012 08:19:05.211] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:05.214] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:05.214] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.215] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:05.226] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@localhost:61382
[fourth] [DEBUG] [06/14/2012 08:19:05.227] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@first
[fourth] [DEBUG] [06/14/2012 08:19:05.230] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[fourth] [INFO] [06/14/2012 08:19:05.232] [main] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@first
[fourth] [DEBUG] [06/14/2012 08:19:05.232] [main] [RemoteClient(akka://ConvergenceSpec)] Sending message Join(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/commands]
[fourth] [DEBUG] [06/14/2012 08:19:05.238] [main] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.ClusterAction$Join]
[fourth] [DEBUG] [06/14/2012 08:19:05.250] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@39edd9b3),CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.250] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.251] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[first] [DEBUG] [06/14/2012 08:19:05.252] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.252] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[first] [INFO] [06/14/2012 08:19:05.252] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@fourth
[first] [DEBUG] [06/14/2012 08:19:05.252] [ConvergenceSpec-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:05.252] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteServerClientConnected@akka://ConvergenceSpec@first: Client[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:05.255] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:05.256] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1e9af0b1),CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.256] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.257] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[first] [DEBUG] [06/14/2012 08:19:05.257] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:05.257] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:05.257] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$h]
[first] [DEBUG] [06/14/2012 08:19:05.257] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:05.257] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:05.257] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:05.257] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:05.257] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.258] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[first] [DEBUG] [06/14/2012 08:19:05.258] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:05.258] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:05.258] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.258] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:05.258] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:05.258] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Join(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/commands]<+{akka://ConvergenceSpec@first/system/cluster/commands} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [INFO] [06/14/2012 08:19:05.258] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Node [akka://ConvergenceSpec@fourth] is JOINING
[fourth] [DEBUG] [06/14/2012 08:19:05.258] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[first] [DEBUG] [06/14/2012 08:19:05.259] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:05.259] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$h])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:05.259] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(second))),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$i]
[first] [DEBUG] [06/14/2012 08:19:05.259] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:05.260] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 14
[first] [DEBUG] [06/14/2012 08:19:05.260] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:05.261] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(GetAddress(RoleName(second)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.261] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:05.261] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(AddressReply(RoleName(second),akka://ConvergenceSpec@second)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:05.261] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.262] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6cf84b0a),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.262] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:05.262] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(second),akka://ConvergenceSpec@second)
[second] [DEBUG] [06/14/2012 08:19:05.262] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[fourth] [DEBUG] [06/14/2012 08:19:05.262] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(second),akka://ConvergenceSpec@second),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Some((second,Actor[akka://ConvergenceSpec/temp/$i])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.262] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.263] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[fourth] [DEBUG] [06/14/2012 08:19:05.263] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(fourth))),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$j]
[fourth] [DEBUG] [06/14/2012 08:19:05.263] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 14
[first] [DEBUG] [06/14/2012 08:19:05.263] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: GetAddress(RoleName(fourth))
[first] [DEBUG] [06/14/2012 08:19:05.263] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(GetAddress(RoleName(fourth)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.263] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(fourth))
[first] [DEBUG] [06/14/2012 08:19:05.264] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)),None) from Actor[akka://ConvergenceSpec/user/controller]
[fourth] [DEBUG] [06/14/2012 08:19:05.264] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)
[fourth] [DEBUG] [06/14/2012 08:19:05.264] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Some((fourth,Actor[akka://ConvergenceSpec/temp/$j])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:05.266] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode4(akka://ConvergenceSpec)] assertNotMovedUp#1
[second] [DEBUG] [06/14/2012 08:19:05.267] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.267] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:05.298] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:05.298] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:05.298] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.298] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4b69d75d),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.298] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.299] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:05.299] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.299] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:05.300] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.300] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.396] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:05.396] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:05.396] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.396] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.396] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:05.397] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:05.398] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.398] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:05.400] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:05.400] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6b1ce2ad),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.400] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.400] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:05.400] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[first] [DEBUG] [06/14/2012 08:19:05.400] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.401] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [ERROR] [06/14/2012 08:19:05.401] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:05.401] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [DEBUG] [06/14/2012 08:19:05.403] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.404] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.405] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.405] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x356602cb] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.405] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:05.406] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2) in direction Send when socket [id: 0x356602cb] already closed, dropping
[second] [INFO] [06/14/2012 08:19:05.406] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:05.406] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:05.455] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:05.456] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:05.456] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:05.456] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:05.456] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:05.456] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:05.457] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:05.457] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.458] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.459] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@519549e),CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.459] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:05.459] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[fourth] [DEBUG] [06/14/2012 08:19:05.459] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:05.460] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=143, cap=143)
[fourth] [DEBUG] [06/14/2012 08:19:05.460] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.460] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[fourth] [DEBUG] [06/14/2012 08:19:05.461] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.461] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:05.461] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=143, cap=143)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.461] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=143, cap=143)
[fourth] [DEBUG] [06/14/2012 08:19:05.493] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:05.514] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [INFO] [06/14/2012 08:19:05.515] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Can't establish a causal relationship between "remote" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))] and "local" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52c56)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52c56))] - merging them into [Gossip(overview = GossipOverview(seen = [], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))]
[fourth] [DEBUG] [06/14/2012 08:19:05.515] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:05.516] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@fourth, status = Joining)]
[fourth] [INFO] [06/14/2012 08:19:05.517] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Leader is moving node [akka://ConvergenceSpec@fourth] from JOINING to UP
[first] [DEBUG] [06/14/2012 08:19:05.554] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:05.554] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.554] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.555] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:05.555] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.555] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@52aa77d9),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.555] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.555] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.555] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:05.556] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3be7a755),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.556] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.556] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.556] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:05.556] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:05.557] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.557] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:05.557] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.557] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:05.559] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:05.559] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:05.593] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:05.594] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:05.594] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.594] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.594] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:05.594] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.598] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [DEBUG] [06/14/2012 08:19:05.596] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.597] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5585c0de),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.597] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.598] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.598] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[first] [DEBUG] [06/14/2012 08:19:05.601] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.601] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:05.610] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:05.610] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:05.610] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.Heartbeat]
[fourth] [DEBUG] [06/14/2012 08:19:05.611] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:05.611] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:05.611] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@12d28dfa),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.612] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:05.611] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:05.612] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.612] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.612] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:05.612] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@second
[second] [DEBUG] [06/14/2012 08:19:05.612] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@localhost:61384
[fourth] [DEBUG] [06/14/2012 08:19:05.613] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.613] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.613] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [INFO] [06/14/2012 08:19:05.613] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:05.613] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:05.613] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@20c3e967),CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.613] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.614] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[fourth] [DEBUG] [06/14/2012 08:19:05.614] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.614] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:05.614] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7844ffb2),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.614] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.614] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.614] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[second] [DEBUG] [06/14/2012 08:19:05.615] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [INFO] [06/14/2012 08:19:05.615] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@fourth
[second] [DEBUG] [06/14/2012 08:19:05.615] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:05.615] [ConvergenceSpec-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:05.615] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteServerClientConnected@akka://ConvergenceSpec@second: Client[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:05.615] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.615] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:05.653] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:05.653] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:05.653] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:05.654] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.656] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.656] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5c6c2308),CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.657] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:05.657] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:05.657] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1167, cap=1167)
[fourth] [DEBUG] [06/14/2012 08:19:05.657] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.658] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:05.658] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=1167, cap=1167)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.658] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1167, cap=1167)
[fourth] [DEBUG] [06/14/2012 08:19:05.661] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:05.662] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:05.687] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:05.687] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:05.687] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:05.687] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[second] [DEBUG] [06/14/2012 08:19:05.693] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:05.693] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.694] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:05.694] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.694] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:05.695] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.695] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.693] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.694] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@12910b0b),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.694] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:05.708] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:05.709] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:05.710] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:05.710] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:05.710] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:05.711] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:05.712] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:05.712] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.GossipEnvelope]
[fourth] [DEBUG] [06/14/2012 08:19:05.719] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3125, cap=3125, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:05.721] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1217e615),CompositeChannelBuffer(ridx=0, widx=3125, cap=3125, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.721] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3125, cap=3125, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.721] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:05.721] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:05.721] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=53, cap=53)
[second] [DEBUG] [06/14/2012 08:19:05.722] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.722] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:05.722] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.722] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:05.722] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=53, cap=53)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.722] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=53, cap=53)
[second] [DEBUG] [06/14/2012 08:19:05.729] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.729] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:05.792] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:05.792] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:05.792] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:05.794] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:05.796] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3221, cap=3221, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.797] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2a44b7f7),CompositeChannelBuffer(ridx=0, widx=3221, cap=3221, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.797] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3221, cap=3221, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:05.797] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[fourth] [DEBUG] [06/14/2012 08:19:05.797] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:05.798] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=149, cap=149)
[fourth] [DEBUG] [06/14/2012 08:19:05.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[fourth] [DEBUG] [06/14/2012 08:19:05.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:05.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=149, cap=149)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.799] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=149, cap=149)
[fourth] [DEBUG] [06/14/2012 08:19:05.807] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:05.808] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.825] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:05.826] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:05.826] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:05.884] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:05.884] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:05.884] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:05.884] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:05.885] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:05.885] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:05.885] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:05.885] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.888] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.888] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@456c1227),CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.888] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.889] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[second] [DEBUG] [06/14/2012 08:19:05.889] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=143, cap=143)
[second] [DEBUG] [06/14/2012 08:19:05.889] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.889] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[second] [DEBUG] [06/14/2012 08:19:05.890] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=143, cap=143)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.890] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=143, cap=143)
[second] [DEBUG] [06/14/2012 08:19:05.893] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52a58, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@third -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a525b6, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.894] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:05.906] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:05.906] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:05.906] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:05.906] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:05.907] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:05.907] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:05.908] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:05.910] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:05.910] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:05.910] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:05.911] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4cd1ad08),CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.911] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.911] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[first] [DEBUG] [06/14/2012 08:19:05.911] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:05.911] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)
[first] [DEBUG] [06/14/2012 08:19:05.911] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.911] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[first] [DEBUG] [06/14/2012 08:19:05.912] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.912] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:05.912] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.912] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.912] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)
[fourth] [DEBUG] [06/14/2012 08:19:05.914] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:05.914] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@38666d83),CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:05.914] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.915] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:05.915] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1233, cap=1233)
[second] [DEBUG] [06/14/2012 08:19:05.915] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.915] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:05.915] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=1233, cap=1233)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.915] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1233, cap=1233)
[second] [DEBUG] [06/14/2012 08:19:05.918] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:05.919] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:05.920] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:05.921] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:05.981] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:05.983] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:05.984] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.984] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.984] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:05.984] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:05.985] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6cb6a34),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.985] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:05.985] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:05.985] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:05.985] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:05.985] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5e1645b9),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.985] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:05.986] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:05.986] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:05.986] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:05.986] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:05.986] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:05.987] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:05.987] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:05.987] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.005] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.006] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.006] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.006] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.006] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.007] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1029f93b),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.007] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.007] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.007] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.007] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@436d4b50),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.007] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.007] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.007] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.008] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:06.008] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.008] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:06.008] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:06.008] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.009] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.009] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.023] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:06.023] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:06.023] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.023] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.024] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.024] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.024] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.026] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.027] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5c6c2308),CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.027] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.027] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:06.028] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1233, cap=1233)
[fourth] [DEBUG] [06/14/2012 08:19:06.028] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.028] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:06.028] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=1233, cap=1233)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.028] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1233, cap=1233)
[fourth] [DEBUG] [06/14/2012 08:19:06.032] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.032] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:06.082] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:06.082] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:06.082] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:06.082] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:06.082] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:06.082] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:06.082] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.083] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:06.087] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.087] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@31734b48),CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.087] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.087] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3317, cap=3317)
[second] [DEBUG] [06/14/2012 08:19:06.088] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3317, cap=3317)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.088] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3317, cap=3317)
[second] [DEBUG] [06/14/2012 08:19:06.092] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.093] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.103] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:06.104] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.104] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.104] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:06.104] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.104] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.105] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.107] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.107] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:06.107] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.107] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@22343bb6),CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.108] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.108] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:06.108] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1233, cap=1233)
[first] [DEBUG] [06/14/2012 08:19:06.108] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.108] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:06.108] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=1233, cap=1233)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.108] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1233, cap=1233)
[fourth] [DEBUG] [06/14/2012 08:19:06.109] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.110] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.111] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1f5726ec),CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.111] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.111] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[second] [DEBUG] [06/14/2012 08:19:06.111] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)
[second] [DEBUG] [06/14/2012 08:19:06.112] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.112] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[second] [DEBUG] [06/14/2012 08:19:06.112] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.112] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)
[first] [DEBUG] [06/14/2012 08:19:06.112] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:06.113] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.128] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.128] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.128] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.128] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.129] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.129] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2310195c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.129] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.129] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.129] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:06.129] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1f6b69d7),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.129] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.129] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.129] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.130] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.130] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.130] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:06.130] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:06.130] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.131] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.131] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.131] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.131] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.221] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:06.222] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:06.222] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.222] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.222] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.222] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.223] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.224] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.225] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:06.225] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.225] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5dbb6a69),CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.225] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.225] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:06.225] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:06.226] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)
[fourth] [DEBUG] [06/14/2012 08:19:06.226] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.226] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:06.226] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.226] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)
[second] [ERROR] [06/14/2012 08:19:06.226] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:06.228] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [DEBUG] [06/14/2012 08:19:06.230] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.230] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.231] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.232] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3b474392] WRITE: CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.232] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2) in direction Send when socket [id: 0x3b474392] already closed, dropping
[second] [DEBUG] [06/14/2012 08:19:06.232] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [INFO] [06/14/2012 08:19:06.232] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:06.232] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:06.280] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:06.280] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:06.280] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:06.280] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:06.280] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:06.281] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.281] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:06.281] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:06.283] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.283] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6588c838),CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.283] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.284] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3317, cap=3317)
[second] [DEBUG] [06/14/2012 08:19:06.284] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3317, cap=3317)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.284] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3317, cap=3317)
[second] [DEBUG] [06/14/2012 08:19:06.288] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.288] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.302] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:06.302] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.302] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.302] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:06.302] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.303] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.303] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.305] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.305] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@26945b95),CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.306] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.306] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:06.306] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.306] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:06.306] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)
[fourth] [DEBUG] [06/14/2012 08:19:06.306] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:06.306] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.306] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:06.306] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.307] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)
[first] [DEBUG] [06/14/2012 08:19:06.310] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [ERROR] [06/14/2012 08:19:06.310] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:06.310] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:06.310] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.311] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.312] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5608a6fc] WRITE: CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.312] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2) in direction Send when socket [id: 0x5608a6fc] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:06.313] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:06.313] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.313] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:06.314] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:06.317] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:06.318] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.320] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3281, cap=3281)
[fourth] [DEBUG] [06/14/2012 08:19:06.319] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.319] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2206270b),CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.319] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.320] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3281, cap=3281)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.320] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3281, cap=3281)
[second] [DEBUG] [06/14/2012 08:19:06.323] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.323] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:06.379] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.380] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:06.380] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.380] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.380] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:06.380] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3f65cebb),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.380] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.381] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.381] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:06.381] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3bc634b9),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.381] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.381] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.381] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.381] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.382] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.382] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:06.382] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.382] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.382] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.383] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.401] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.402] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.402] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.402] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.402] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.402] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@144f3ba2),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.403] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.403] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.403] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.403] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@e31bcc8),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.403] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.403] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.403] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:06.403] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:06.403] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.404] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:06.404] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:06.404] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.404] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.404] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.419] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:06.419] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:06.419] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.419] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.419] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.420] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.422] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[second] [DEBUG] [06/14/2012 08:19:06.420] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.422] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.422] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5704a4b6),CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.422] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3281, cap=3281, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.423] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)
[first] [DEBUG] [06/14/2012 08:19:06.423] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.423] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:06.423] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.423] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=209, cap=209)
[first] [DEBUG] [06/14/2012 08:19:06.426] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:06.427] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:06.478] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:06.478] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:06.478] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:06.478] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:06.478] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:06.479] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:06.479] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.480] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:06.481] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.482] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5c17d6c1),CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.482] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.482] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:06.482] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=245, cap=245)
[fourth] [DEBUG] [06/14/2012 08:19:06.482] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.482] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:06.482] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=245, cap=245)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.483] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=245, cap=245)
[fourth] [DEBUG] [06/14/2012 08:19:06.486] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.486] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.500] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:06.500] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.500] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:06.500] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.500] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.500] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.501] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.503] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.503] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@50739aa3),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.503] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.503] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:06.504] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.504] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:06.507] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.507] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.518] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.519] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.519] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.519] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.519] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.520] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:06.520] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:06.520] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.520] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:06.521] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:06.521] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:06.520] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.520] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:06.519] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5712bd54),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.519] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.520] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.520] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4735572b),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.520] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.521] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.521] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.580] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:06.581] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[second] [DEBUG] [06/14/2012 08:19:06.617] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:06.617] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:06.617] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.617] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.617] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.617] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.622] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:06.619] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.621] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.621] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:06.621] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.621] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5e28df48),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.621] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.621] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[second] [ERROR] [06/14/2012 08:19:06.622] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:06.623] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:06.622] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.622] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:06.625] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.626] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.624] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.625] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4d5b2043] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.627] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x4d5b2043] already closed, dropping
[second] [DEBUG] [06/14/2012 08:19:06.627] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [INFO] [06/14/2012 08:19:06.627] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:06.627] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:06.676] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:06.676] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:06.676] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:06.676] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:06.677] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:06.677] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:06.677] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.677] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:06.678] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.679] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@78304648),CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.679] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.679] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:06.679] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.679] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3317, cap=3317)
[fourth] [DEBUG] [06/14/2012 08:19:06.680] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3317, cap=3317)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.680] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3317, cap=3317)
[first] [DEBUG] [06/14/2012 08:19:06.682] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:06.683] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.683] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [ERROR] [06/14/2012 08:19:06.689] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:06.689] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:06.690] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:06.691] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11cdc190] WRITE: CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.691] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3317, cap=3317, components=2) in direction Send when socket [id: 0x11cdc190] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:06.692] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:06.692] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:06.695] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:06.698] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:06.698] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.698] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:06.698] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.698] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.698] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.699] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.702] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.702] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:06.704] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.704] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6c98a8b8),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.704] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.704] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[second] [DEBUG] [06/14/2012 08:19:06.704] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:06.705] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.705] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [ERROR] [06/14/2012 08:19:06.706] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:06.706] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:06.707] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.708] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1a6cffbb] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.708] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.709] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x1a6cffbb] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:06.709] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:06.709] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:06.709] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.709] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:06.776] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.776] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:06.777] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.777] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.778] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:06.778] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1e35ecd),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.778] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.779] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.779] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:06.779] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5a2445d7),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.779] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.779] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.779] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.780] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.780] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.781] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:06.781] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.781] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.782] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.782] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.797] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.797] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.798] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.798] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.798] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.799] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@677456bc),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.799] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.799] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.799] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.799] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@eefeff8),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.799] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.800] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.800] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:06.800] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:06.800] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.800] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:06.801] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:06.801] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.801] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.801] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.815] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:06.816] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:06.816] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.816] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.816] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:06.816] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.817] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.818] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.818] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:06.819] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.819] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:06.819] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.820] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:06.819] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@73feb3a6),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.819] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.819] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[second] [ERROR] [06/14/2012 08:19:06.821] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:06.821] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [DEBUG] [06/14/2012 08:19:06.822] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:06.824] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:06.824] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.824] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x76e9493a] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.825] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x76e9493a] already closed, dropping
[second] [DEBUG] [06/14/2012 08:19:06.825] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:06.825] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@fourth]
[second] [INFO] [06/14/2012 08:19:06.825] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:06.826] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.826] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[second] [DEBUG] [06/14/2012 08:19:06.826] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.828] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.828] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@37d3ac6e),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.828] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.828] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:06.828] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.829] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:06.832] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.832] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:06.874] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:06.874] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:06.874] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:06.874] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:06.875] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:06.875] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:06.875] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:06.876] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:06.878] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.879] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@b83d59c),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.879] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.879] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:06.879] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.879] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:06.882] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.883] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.896] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:06.896] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:06.897] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:06.897] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.897] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.898] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:06.899] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:06.903] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:06.903] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@caa559d),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.903] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.904] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:06.904] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.904] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:06.909] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:06.909] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:06.915] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:06.915] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.916] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:06.916] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:06.917] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:06.917] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.914] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.914] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.914] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.915] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:06.916] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:06.916] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:06.916] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:06.914] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:06.915] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:06.915] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@ea224b5),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.915] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.915] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:06.915] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@43052ce),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:06.915] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:06.982] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode1(akka://ConvergenceSpec)] assertNotMovedUp#2
[first] [DEBUG] [06/14/2012 08:19:06.984] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[second] [DEBUG] [06/14/2012 08:19:07.013] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:07.013] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:07.013] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:07.013] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.014] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.014] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.014] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.016] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.016] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:07.016] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.016] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@cb80017),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.016] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:07.016] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.023] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.016] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [ERROR] [06/14/2012 08:19:07.017] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:07.017] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:07.023] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.023] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.027] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.028] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:07.017] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.017] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:07.021] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:07.022] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:07.018] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.020] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x040dec87] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.020] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x040dec87] already closed, dropping
[second] [DEBUG] [06/14/2012 08:19:07.020] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.020] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [INFO] [06/14/2012 08:19:07.020] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:07.020] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.020] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[second] [DEBUG] [06/14/2012 08:19:07.021] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.022] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.022] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5cee644c),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.022] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.072] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:07.072] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:07.072] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:07.072] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:07.072] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.072] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.072] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.073] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.074] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.075] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@e7c0d41),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.075] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.075] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.075] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.075] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.078] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.079] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.094] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:07.095] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.095] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:07.095] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:07.095] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:07.095] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.096] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.097] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.097] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2755d4be),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.098] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.098] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.098] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.098] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.102] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.102] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.171] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.171] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.171] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.172] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.172] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.172] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6406f3ea),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.172] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.172] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.172] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5092eb76),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.172] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:07.173] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.173] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.173] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.173] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.173] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.173] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:07.174] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.174] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.174] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.174] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.193] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.193] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.193] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.193] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.193] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.193] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3600e312),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.194] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.194] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.194] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.194] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@19fcbac1),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.194] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.194] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.194] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:07.194] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:07.194] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.195] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:07.195] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.195] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:07.195] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.195] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:07.211] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:07.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.211] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:07.211] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.214] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.212] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.213] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.213] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:07.214] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.214] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:07.214] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.215] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.214] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@447cc9c9),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.214] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [ERROR] [06/14/2012 08:19:07.214] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:07.215] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [DEBUG] [06/14/2012 08:19:07.216] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.218] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.217] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5ca3ce3f] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.219] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x5ca3ce3f] already closed, dropping
[second] [DEBUG] [06/14/2012 08:19:07.219] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [INFO] [06/14/2012 08:19:07.219] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:07.219] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:07.219] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:07.271] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:07.271] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:07.271] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:07.271] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.271] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.271] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:07.271] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.272] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.273] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.273] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:07.273] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.273] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@57044c5),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.273] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.274] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[second] [DEBUG] [06/14/2012 08:19:07.274] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.274] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.274] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [ERROR] [06/14/2012 08:19:07.274] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:07.275] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:07.276] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.278] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0db2e44d] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.278] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:07.279] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x0db2e44d] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:07.279] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:07.279] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:07.279] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:07.279] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:07.292] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:07.292] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.292] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:07.292] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:07.292] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:07.292] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.293] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.295] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.295] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:07.295] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.295] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@58b45130),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.295] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.295] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:07.296] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=305, cap=305)
[first] [DEBUG] [06/14/2012 08:19:07.296] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.296] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:07.296] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=305, cap=305)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.296] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=305, cap=305)
[fourth] [DEBUG] [06/14/2012 08:19:07.296] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.298] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.298] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@23bb9829),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.299] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.299] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.299] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.299] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:07.300] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:07.300] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.302] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.303] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.310] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.310] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.311] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.311] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.311] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.311] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2c7103af),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.312] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:07.311] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.312] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.311] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.311] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.311] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7b5c9f9d),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.312] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.312] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.313] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.313] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:07.312] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.312] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:07.313] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:07.313] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:07.409] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:07.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:07.409] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:07.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.409] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.410] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.412] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.412] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:07.412] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.412] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@67bf6ec8),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.412] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.412] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:07.412] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:07.413] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.413] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [ERROR] [06/14/2012 08:19:07.413] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:07.414] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [DEBUG] [06/14/2012 08:19:07.415] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.416] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x490241ae] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.416] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x490241ae] already closed, dropping
[second] [DEBUG] [06/14/2012 08:19:07.416] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:07.416] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@fourth]
[second] [INFO] [06/14/2012 08:19:07.417] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:07.417] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.417] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:07.417] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:07.417] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:07.418] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.419] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.419] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3f4a0fe1),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.419] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.420] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.420] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.420] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.424] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.424] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:07.469] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:07.469] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:07.469] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:07.470] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.470] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.470] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.471] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:07.472] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.474] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.474] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7c6743e2),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.475] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.475] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.476] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.476] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.480] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.481] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.490] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:07.490] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.490] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:07.490] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:07.490] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:07.491] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.491] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.493] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.493] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:07.493] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.493] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2beb717e),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.493] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.493] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[second] [DEBUG] [06/14/2012 08:19:07.493] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.494] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.494] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [ERROR] [06/14/2012 08:19:07.494] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:07.494] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:07.495] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.496] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6e681db8] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.497] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x6e681db8] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:07.497] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:07.497] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:07.497] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[second] [DEBUG] [06/14/2012 08:19:07.497] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.498] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.567] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.567] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.568] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.568] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.569] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.569] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@71e13a2c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.569] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.570] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.570] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:07.570] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2bf87baf),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.570] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.570] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.570] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.570] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.571] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.571] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:07.571] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.572] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.572] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.573] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:07.581] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode2(akka://ConvergenceSpec)] assertNotMovedUp#2
[second] [DEBUG] [06/14/2012 08:19:07.582] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:07.583] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode4(akka://ConvergenceSpec)] assertNotMovedUp#2
[fourth] [DEBUG] [06/14/2012 08:19:07.584] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:07.589] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.589] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.590] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.590] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.590] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.590] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@76c7cadf),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.590] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.590] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.591] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.591] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2c164804),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.591] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.591] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.591] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:07.591] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:07.591] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.591] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:07.592] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:07.592] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.592] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.592] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.607] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:07.607] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:07.608] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:07.608] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.608] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.608] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.611] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.609] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.610] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.611] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@45de530a),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.611] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.611] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.611] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.615] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.615] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:07.666] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:07.667] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:07.667] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:07.667] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.667] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.667] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:07.667] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.668] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.669] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.669] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@68487fb7),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.669] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.670] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.670] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.670] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.673] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.673] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.688] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:07.688] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.688] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:07.688] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:07.688] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:07.688] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.689] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.690] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.691] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@251c135c),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.691] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.691] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.691] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.691] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.694] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.694] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.706] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.706] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.706] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.706] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.706] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.706] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@71dd95c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.707] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:07.707] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.707] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.707] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.708] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.708] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:07.707] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.707] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.707] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6a301bf5),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.707] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.707] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.707] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:07.708] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:07.708] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:07.805] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:07.805] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:07.805] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:07.805] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.806] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.806] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:07.806] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:07.807] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.808] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3bdbe135),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.808] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.808] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.808] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.808] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.812] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.812] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:07.865] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:07.865] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:07.865] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.865] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:07.865] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.865] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.865] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:07.866] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.867] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.867] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@db2b10a),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.867] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.868] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.868] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.868] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:07.871] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.872] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.885] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:07.886] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.886] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:07.886] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:07.886] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:07.886] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.887] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.888] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.889] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@77b9e7fc),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.889] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.889] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.889] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.889] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:07.893] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.893] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:07.963] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.963] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.963] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.963] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:07.964] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.964] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@17e319a),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.964] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.964] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.964] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@15f7dbd8),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.964] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.964] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:07.965] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.965] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:07.965] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.965] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.965] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:07.965] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.965] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:07.966] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:07.966] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:07.984] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode1(akka://ConvergenceSpec)] assertNotMovedUp#3
[fourth] [DEBUG] [06/14/2012 08:19:07.985] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.985] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:07.985] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:07.985] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:07.985] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:07.986] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:07.986] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@10fa1b2d),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.986] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.986] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.986] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:07.986] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:07.986] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@48423606),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:07.986] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:07.986] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:07.987] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:07.987] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:07.987] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:07.987] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:07.987] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:07.988] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:07.988] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.003] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:08.003] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:08.003] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.003] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.003] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.003] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:08.006] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.004] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.005] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.005] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.005] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.005] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5e15e68d),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.006] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.006] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.006] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.006] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.008] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.009] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@360b0c5a),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.009] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.009] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:08.009] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.009] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:08.011] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:08.011] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:08.014] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:08.014] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:08.062] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:08.063] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:08.063] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:08.063] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.063] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.063] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:08.063] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:08.064] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:08.065] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.065] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@615724a0),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.065] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.066] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.066] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.066] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.068] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.069] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.085] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:08.085] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.085] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:08.085] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:08.085] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:08.085] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.086] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.087] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.088] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4f13f501),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.088] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.088] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.088] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.088] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.091] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.091] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.102] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.102] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.103] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.103] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.103] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.103] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@f58046e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.104] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.103] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:08.104] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.104] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:08.105] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:08.105] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.103] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.103] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.103] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@78d5c653),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.103] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.104] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.104] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.105] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:08.105] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.201] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:08.201] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:08.202] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.202] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.202] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.203] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.204] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.206] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.206] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:08.206] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.207] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@dad4feb),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.207] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.207] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:08.207] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:08.208] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.208] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [ERROR] [06/14/2012 08:19:08.208] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:08.208] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [DEBUG] [06/14/2012 08:19:08.210] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.213] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6d59d450] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.213] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x6d59d450] already closed, dropping
[second] [DEBUG] [06/14/2012 08:19:08.213] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:08.213] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@fourth]
[second] [INFO] [06/14/2012 08:19:08.213] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:08.214] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[second] [DEBUG] [06/14/2012 08:19:08.214] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.214] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:08.214] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:08.215] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.215] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.216] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@64cbbe37),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.216] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.216] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:08.216] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.216] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:08.219] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:08.220] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:08.259] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:08.260] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.260] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:08.260] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:08.260] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.260] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:08.260] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:08.261] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:08.262] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.263] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@f8b7298),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.263] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.263] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:08.263] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.264] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:08.267] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:08.267] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.282] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:08.282] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.282] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:08.282] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:08.282] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:08.282] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.283] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.284] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.285] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2caee320),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.285] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.285] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.285] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.285] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.288] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.288] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.360] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:08.360] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:08.360] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.361] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:08.361] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:08.361] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@758c3b7),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.362] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.362] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.362] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.362] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.362] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6b1af7bb),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.363] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.363] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.363] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:08.364] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.364] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:08.364] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.364] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.365] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:08.365] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.381] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.381] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.381] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.381] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.382] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.382] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@12b9b67b),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.382] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.382] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.382] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.383] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.383] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.383] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:08.383] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:08.383] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@32935741),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.384] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.384] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:08.384] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.384] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:08.385] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.385] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.399] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:08.399] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.399] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:08.399] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.399] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.399] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.405] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:08.402] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:08.405] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.405] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.400] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.401] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.401] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.401] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:08.402] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.402] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.401] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7595ddb5),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.406] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:08.407] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.401] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.402] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.404] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.404] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@123aabc1),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.404] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.408] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:08.409] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:08.458] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:08.458] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:08.458] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:08.458] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.458] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.458] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:08.458] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:08.459] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:08.460] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.461] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5852f73e),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.461] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.461] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.462] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.462] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.464] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.465] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.480] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:08.480] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:08.480] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.481] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.482] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.483] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:08.483] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.483] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@13ceebfa),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.483] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.483] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[second] [DEBUG] [06/14/2012 08:19:08.483] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.484] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.484] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [ERROR] [06/14/2012 08:19:08.484] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:08.484] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:08.485] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.487] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.487] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.498] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.498] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.498] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.498] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.499] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.499] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@49e21540),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.495] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6cd9ceea] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.499] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:08.500] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x6cd9ceea] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:08.500] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:08.499] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.499] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.499] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@13de6be9),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.499] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [INFO] [06/14/2012 08:19:08.500] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:08.500] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:08.501] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:08.501] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.501] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.499] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.499] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.500] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:08.500] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:08.501] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:08.501] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:08.513] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.514] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:08.514] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.582] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode2(akka://ConvergenceSpec)] assertNotMovedUp#3
[fourth] [DEBUG] [06/14/2012 08:19:08.585] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode4(akka://ConvergenceSpec)] assertNotMovedUp#3
[second] [DEBUG] [06/14/2012 08:19:08.585] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:08.587] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[second] [DEBUG] [06/14/2012 08:19:08.597] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:08.598] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:08.598] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.598] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.598] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.598] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.600] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.602] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.602] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:08.602] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.602] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6bb56848),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.602] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.602] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:08.603] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:08.603] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.603] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [ERROR] [06/14/2012 08:19:08.603] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:08.604] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [DEBUG] [06/14/2012 08:19:08.605] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.606] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x52d18636] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.607] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x52d18636] already closed, dropping
[second] [DEBUG] [06/14/2012 08:19:08.607] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [INFO] [06/14/2012 08:19:08.607] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:08.607] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:08.608] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:08.609] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:08.655] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:08.656] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:08.656] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:08.656] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.656] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.656] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:08.656] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:08.657] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:08.658] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.658] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@22cb4138),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.658] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.659] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.659] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.659] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.662] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.662] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.711] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:08.711] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.711] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:08.712] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:08.712] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:08.712] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.713] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.714] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.714] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:08.714] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.714] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4e4b35ef),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.714] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.715] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:08.715] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.715] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:08.715] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.716] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.716] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4eff0d4b),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.717] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.717] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.717] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.717] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:08.720] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:08.720] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.721] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.721] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.755] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:08.755] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:08.755] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.755] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:08.756] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:08.756] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@18f4a376),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.756] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.756] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.756] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.756] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7342f703),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.756] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.756] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.756] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:08.756] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:08.757] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.757] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:08.757] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.757] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.757] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:08.758] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:08.795] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:08.795] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:08.795] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.795] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.795] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.795] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.798] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:08.798] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.796] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.798] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:08.801] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:08.802] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.797] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.797] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@316becfe),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.797] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.810] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.810] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.810] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.810] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.810] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.810] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4ac1df71),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.810] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.811] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.811] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2f6e4ddd),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.811] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.811] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.811] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.811] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.811] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:08.811] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.811] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.812] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:08.812] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.812] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.812] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.854] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:08.855] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:08.855] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:08.855] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.855] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:08.855] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:08.855] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:08.855] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:08.857] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.857] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7bb03ee),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.857] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.857] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.857] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.857] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.860] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:08.860] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:08.894] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.895] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.895] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.895] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.895] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.895] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2494310b),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.896] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.896] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:08.896] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.896] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:08.897] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:08.897] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.895] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.895] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.896] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@36a06816),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.896] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.896] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.896] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:08.897] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:08.897] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:08.909] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:08.909] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:08.909] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:08.909] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:08.909] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:08.909] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.910] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.911] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.912] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:08.912] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:08.912] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@45ee1775),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:08.912] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.912] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:08.912] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:08.913] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.913] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [ERROR] [06/14/2012 08:19:08.913] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:08.913] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:08.914] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:08.916] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x12462631] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:08.916] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x12462631] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:08.916] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:08.916] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [INFO] [06/14/2012 08:19:08.916] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:08.917] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:08.916] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:08.986] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode1(akka://ConvergenceSpec)] assertNotMovedUp#4
[first] [DEBUG] [06/14/2012 08:19:08.988] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[second] [DEBUG] [06/14/2012 08:19:08.993] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:08.993] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:08.993] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.994] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:08.994] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.994] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.995] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:08.997] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:08.997] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:08.997] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:08.997] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@44f757b9),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:08.998] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:08.998] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:08.998] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:08.998] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:08.998] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.000] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.000] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@781fb069),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.001] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.001] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.001] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.001] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:09.003] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:09.004] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:09.005] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.005] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:09.085] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:09.085] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.085] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:09.086] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:09.086] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.086] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:09.086] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.086] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:09.087] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.088] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@17bf9b45),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.088] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.088] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.088] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.088] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.091] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.092] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.107] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:09.107] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.107] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:09.107] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:09.107] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:09.107] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.108] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.109] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.109] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:09.109] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.110] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@37c3a0c7),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.110] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.110] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[second] [DEBUG] [06/14/2012 08:19:09.110] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:09.110] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.110] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [ERROR] [06/14/2012 08:19:09.110] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:09.111] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:09.111] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.113] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0c70856b] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.113] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x0c70856b] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:09.113] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:09.113] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:09.113] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[second] [DEBUG] [06/14/2012 08:19:09.113] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:09.114] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.184] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.185] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:09.185] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.186] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.186] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:09.186] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@635c80a4),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.186] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.187] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.187] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:09.187] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@55641ee0),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.187] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.187] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.187] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.187] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.188] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.188] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:09.189] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:09.189] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.190] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.190] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.207] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.207] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.207] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.207] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.208] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.208] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@8f5cbda),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.208] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.208] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.208] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@28fe04c6),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.208] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.209] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.209] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.209] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:09.209] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:09.209] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.209] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:09.210] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:09.210] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.210] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:09.211] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.223] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:09.224] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:09.224] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.224] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:09.224] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.224] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.227] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:09.227] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.227] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.230] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.230] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.230] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:09.231] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:09.232] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:09.225] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.226] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.226] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.226] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:09.226] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3be67280),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.226] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.234] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.234] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:09.227] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.229] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.229] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2207d8bb),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.230] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.283] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:09.283] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:09.284] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:09.284] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.284] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.284] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:09.284] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.284] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:09.286] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.286] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@662ad0d),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.286] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.286] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.286] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.287] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.289] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.290] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.305] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:09.305] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.305] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:09.305] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:09.305] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:09.306] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.306] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.307] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.308] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:09.308] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.308] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7ec71c00),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.308] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.308] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:09.308] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.309] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.309] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.310] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.310] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2161fcdd),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.311] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.311] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:09.311] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.311] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:09.312] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:09.312] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.314] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:09.315] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.323] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:09.323] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.323] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.323] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:09.323] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3125ee71),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.324] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.324] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.324] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.324] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:09.324] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@45a8123b),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.324] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.324] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.324] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.324] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.325] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.325] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:09.325] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:09.325] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:09.325] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.325] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:09.422] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:09.423] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:09.423] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:09.423] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.424] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.424] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:09.425] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.427] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.427] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:09.428] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:09.428] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1fe91485),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.428] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[second] [DEBUG] [06/14/2012 08:19:09.428] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.428] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:09.429] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.429] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [ERROR] [06/14/2012 08:19:09.429] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:09.430] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [DEBUG] [06/14/2012 08:19:09.431] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.433] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7f423820] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.433] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x7f423820] already closed, dropping
[second] [DEBUG] [06/14/2012 08:19:09.433] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [INFO] [06/14/2012 08:19:09.433] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:09.433] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:09.435] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:09.435] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:09.481] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:09.481] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:09.481] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.481] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:09.482] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.482] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.482] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:09.482] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:09.483] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.484] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:09.484] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.484] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@60c0a150),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.484] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.484] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:09.484] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.484] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.484] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [ERROR] [06/14/2012 08:19:09.485] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:09.486] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:09.487] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.488] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.489] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:09.489] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x069ebcd0] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.489] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x069ebcd0] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:09.489] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:09.489] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:09.489] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:09.503] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:09.503] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.504] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:09.504] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:09.504] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:09.505] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.506] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.508] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.508] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:09.509] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.509] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2af4fb44),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.509] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.509] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[second] [DEBUG] [06/14/2012 08:19:09.509] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:09.510] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.510] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [ERROR] [06/14/2012 08:19:09.510] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:09.511] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:09.512] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.514] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x399c4364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.515] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x399c4364] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:09.515] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:09.515] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:09.516] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[second] [DEBUG] [06/14/2012 08:19:09.517] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:09.517] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.580] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.580] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:09.580] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.581] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.581] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:09.581] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@864dfeb),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.581] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.581] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.581] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@213eb2bc),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.581] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:09.581] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.582] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.582] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.582] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.582] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.582] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:09.582] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:09.583] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.583] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.583] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:09.585] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode2(akka://ConvergenceSpec)] assertNotMovedUp#4
[second] [DEBUG] [06/14/2012 08:19:09.586] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:09.587] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode4(akka://ConvergenceSpec)] assertNotMovedUp#4
[fourth] [DEBUG] [06/14/2012 08:19:09.588] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:09.602] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.603] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.603] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.604] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.604] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7b537060),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.604] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.604] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.605] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.605] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.605] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@17b60b6),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.605] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.605] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.605] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:09.605] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:09.606] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.606] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:09.607] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:09.607] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:09.607] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.607] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.620] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:09.620] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:09.620] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:09.620] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.621] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.621] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.631] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:09.623] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.631] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.631] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:09.621] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.623] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.623] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:09.623] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@315e4dbe),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.623] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:09.623] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.624] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.623] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[second] [ERROR] [06/14/2012 08:19:09.624] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:09.629] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:09.626] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [ERROR] [06/14/2012 08:19:09.626] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [DEBUG] [06/14/2012 08:19:09.627] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x26420e44] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.627] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x26420e44] already closed, dropping
[second] [DEBUG] [06/14/2012 08:19:09.627] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:09.627] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@fourth]
[second] [INFO] [06/14/2012 08:19:09.628] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:09.628] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:09.628] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[second] [DEBUG] [06/14/2012 08:19:09.628] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.630] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.631] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1a1a7aa4),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.631] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.634] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.635] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:09.638] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:09.639] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:09.679] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:09.679] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:09.679] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:09.679] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.679] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.679] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:09.679] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.680] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:09.681] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.682] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7f85e155),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.682] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.682] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.682] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.682] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.685] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.686] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.701] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:09.701] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.701] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:09.701] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:09.701] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:09.701] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.702] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.703] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.703] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:09.703] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@57ab4292),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.703] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.703] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.704] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:09.704] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.704] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:09.704] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.706] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.706] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7981c4eb),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.706] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.706] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:09.706] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.706] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:09.707] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:09.707] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.709] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:09.710] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.719] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:09.720] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.720] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.720] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:09.720] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.720] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4069b25f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.721] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:09.721] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.721] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.721] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.722] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.722] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:09.720] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.720] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.720] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@29cd761a),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.721] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.721] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.721] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:09.722] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:09.722] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:09.818] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:09.818] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:09.818] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:09.818] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.818] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:09.818] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:09.819] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:09.820] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.820] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@e9576fd),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.820] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.820] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:09.820] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.820] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:09.823] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:09.823] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:09.877] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:09.877] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:09.877] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:09.877] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.877] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.878] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:09.878] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.878] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:09.879] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.880] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@b5c44d6),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.880] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.880] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:09.880] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.880] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:09.883] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:09.883] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.899] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:09.899] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.899] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:09.899] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:09.900] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:09.900] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.900] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.901] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.902] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@55611ed3),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.902] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.902] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:09.902] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.902] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:09.905] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:09.905] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:09.976] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.977] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:09.977] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.978] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:09.978] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:09.978] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@56ab436d),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.978] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.978] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.978] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1d05a248),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:09.978] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:09.979] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:09.979] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.979] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:09.979] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.980] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.980] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:09.980] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:09.980] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:09.981] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:09.981] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:09.988] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode1(akka://ConvergenceSpec)] assertNotMovedUp#5
[first] [DEBUG] [06/14/2012 08:19:09.990] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:09.998] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.998] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.998] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.998] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:09.998] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:09.999] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5b2eda6f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.999] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.999] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:09.999] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:09.999] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@961401f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:09.999] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:09.999] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:09.999] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:09.999] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:10.000] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.000] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:10.000] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:10.000] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.000] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.001] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.016] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:10.016] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:10.016] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:10.016] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.016] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.016] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:10.018] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.017] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:10.019] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.019] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.018] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.022] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.018] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1d256a73),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.018] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.022] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.075] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:10.076] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:10.076] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.076] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.076] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.076] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:10.076] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:10.077] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:10.078] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.078] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@ba3bff5),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.078] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.078] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.079] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.079] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.082] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.083] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.097] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:10.097] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.097] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:10.097] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:10.097] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:10.097] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.098] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.099] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.099] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4f27077b),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.099] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.100] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.100] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.100] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.102] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.103] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.115] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:10.115] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:10.116] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.116] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:10.116] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:10.116] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@203ba002),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.116] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:10.116] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:10.117] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.117] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:10.118] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:10.118] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:10.116] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.116] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.116] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@404b7c69),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.116] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.117] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.117] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:10.117] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:10.117] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:10.214] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:10.215] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:10.215] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:10.216] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.216] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.216] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:10.217] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:10.219] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.219] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4fee225),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.219] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.220] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.220] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.220] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.225] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:10.225] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.273] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:10.273] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:10.273] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.273] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.273] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.273] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:10.273] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:10.274] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:10.275] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.275] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4f7820c3),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.275] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.275] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.276] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.276] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.278] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:10.279] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.295] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:10.295] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.295] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:10.295] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:10.295] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:10.295] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.296] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.297] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.297] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:10.297] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.297] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@27cb01e3),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.298] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.298] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:10.298] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.298] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.298] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.299] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.299] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@37af78ce),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.300] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.300] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.300] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.300] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:10.302] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:10.302] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.303] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.303] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.372] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:10.372] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:10.373] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.373] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:10.373] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:10.373] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@57af0af7),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.373] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.373] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.373] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5a47eaec),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.373] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.373] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:10.373] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.374] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:10.374] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:10.374] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.374] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:10.374] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.374] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.375] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:10.375] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.394] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.394] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.395] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.395] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.395] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.395] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@612438f1),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.395] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.395] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.395] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5e3d5149),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.395] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:10.395] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.396] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:10.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:10.396] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.396] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:10.396] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:10.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.397] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.397] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.412] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:10.412] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.412] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:10.412] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:10.412] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.413] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:10.413] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:10.414] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.414] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.414] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:10.414] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@72dd23cf),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.414] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.417] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.417] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.417] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:10.415] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.415] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:10.416] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.416] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@21ff3fcf),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.416] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.415] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.415] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:10.419] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:10.419] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:10.421] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:10.422] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.471] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:10.471] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:10.471] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.471] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.471] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.471] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:10.471] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:10.472] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:10.473] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.473] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@246af18d),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.473] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.473] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.474] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.474] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.476] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.476] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.493] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:10.493] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.493] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:10.493] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:10.493] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:10.493] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.494] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.495] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.495] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3ff5d699),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.495] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.496] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.496] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.496] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.498] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.498] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.512] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:10.512] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:10.512] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.512] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:10.512] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:10.512] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7cb64078),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.513] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:10.513] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:10.513] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.513] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:10.514] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:10.514] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:10.512] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.512] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.513] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@26e56328),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.513] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.513] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.513] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:10.514] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:10.514] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:10.586] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode2(akka://ConvergenceSpec)] assertNotMovedUp#5
[second] [DEBUG] [06/14/2012 08:19:10.587] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:10.589] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode4(akka://ConvergenceSpec)] assertNotMovedUp#5
[fourth] [DEBUG] [06/14/2012 08:19:10.590] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[second] [DEBUG] [06/14/2012 08:19:10.610] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:10.610] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.610] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:10.611] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:10.611] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.611] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:10.611] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:10.612] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.612] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6135b95d),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.612] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.613] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.613] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.613] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.616] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:10.617] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.669] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:10.669] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:10.669] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.669] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.669] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.669] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:10.669] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:10.670] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:10.671] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.671] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@22862041),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.671] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.672] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.672] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.672] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.674] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.675] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.691] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:10.691] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.691] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:10.691] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:10.691] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:10.691] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.692] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.693] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.693] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:10.693] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.693] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5ad89e02),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.694] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.694] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:10.694] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.694] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.694] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.695] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.696] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3da1a93),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.696] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.696] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.696] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.696] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:10.697] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:10.698] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.699] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.699] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.768] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:10.768] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:10.769] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.769] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:10.769] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:10.769] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@13af9d1c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.769] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.769] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.769] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@69ea25aa),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.769] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:10.769] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.770] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.770] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:10.770] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:10.770] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.770] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:10.770] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.771] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.771] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:10.771] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.790] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.790] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.790] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.790] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.790] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.791] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6406aade),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.791] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.791] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.791] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:10.791] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6fb9658e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.791] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.791] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.791] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:10.791] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:10.792] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.792] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:10.792] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:10.792] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.792] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.793] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.808] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:10.808] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:10.808] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:10.808] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.808] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.808] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.810] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:10.809] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:10.810] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.810] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@308ab721),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.810] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.810] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.810] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.813] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:10.813] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.867] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:10.867] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:10.867] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.867] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:10.867] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.867] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:10.867] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:10.868] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:10.869] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.869] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7d6d4e3e),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.870] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.870] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.870] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.870] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:10.873] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:10.873] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.889] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:10.890] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:10.890] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:10.890] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:10.890] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:10.890] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:10.891] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.892] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.892] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@261ca3cb),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.892] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:10.892] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:10.892] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.893] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:10.895] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:10.895] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:10.907] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:10.907] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:10.908] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.908] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:10.908] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:10.909] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:10.908] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:10.909] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:10.909] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:10.910] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:10.908] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@62fcf06c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.908] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.908] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:10.908] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@45e41830),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:10.908] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:10.910] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.909] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:10.909] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:10.909] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:10.910] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:10.990] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-3)
[first] [DEBUG] [06/14/2012 08:19:10.991] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-3)),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$m]
[first] [DEBUG] [06/14/2012 08:19:10.991] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:10.991] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61368: EnterBarrier(after-3)
[first] [DEBUG] [06/14/2012 08:19:10.991] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(EnterBarrier(after-3),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:10.991] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-3)
[first] [DEBUG] [06/14/2012 08:19:10.991] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-3),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]
[first] [DEBUG] [06/14/2012 08:19:10.991] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] setting timer 'Timeout'/30000 milliseconds: StateTimeout
[first] [DEBUG] [06/14/2012 08:19:10.992] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] transition Idle -> Waiting
[second] [DEBUG] [06/14/2012 08:19:11.006] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:11.006] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:11.006] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:11.006] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.006] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.006] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:11.007] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:11.007] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.008] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5e1387c6),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.008] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.008] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:11.008] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.008] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:11.011] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:11.011] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:11.065] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:11.065] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:11.065] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:11.065] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:11.065] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:11.065] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:11.065] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:11.066] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:11.067] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.067] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:11.067] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1e05b891),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.067] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:11.068] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.068] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:11.068] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:11.068] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.068] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [ERROR] [06/14/2012 08:19:11.068] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:11.069] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:11.069] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:11.071] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:11.071] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x099846fd] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.071] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:11.071] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x099846fd] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:11.071] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:11.071] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:11.072] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:11.087] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:11.087] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:11.088] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:11.088] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:11.088] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:11.088] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:11.088] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:11.090] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.090] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:11.090] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:11.090] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@127208e4),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.090] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.090] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:11.090] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:11.091] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.091] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [ERROR] [06/14/2012 08:19:11.091] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:11.091] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:11.092] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:11.094] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:11.094] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:11.094] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7ca5ba31] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.095] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2) in direction Send when socket [id: 0x7ca5ba31] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:11.095] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:11.095] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:11.095] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:11.095] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:11.096] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:11.096] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:11.097] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.097] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@17a7905e),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.097] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.098] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:11.098] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.098] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:11.100] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:11.101] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:11.164] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:11.164] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:11.164] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.165] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:11.165] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:11.165] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5e8e918a),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.165] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.165] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.165] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@26fc5b01),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.165] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.165] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:11.165] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.166] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:11.165] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:11.166] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.166] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:11.166] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:11.166] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:11.167] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:11.167] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:11.186] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:11.186] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:11.186] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.186] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:11.186] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:11.186] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@79f71773),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.187] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.187] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.187] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@271a2576),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.187] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.187] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:11.187] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:11.187] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.187] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:11.187] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.187] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:11.188] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:11.188] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.188] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:11.188] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.204] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:11.204] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:11.205] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:11.205] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.205] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.205] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:11.207] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:11.205] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:11.206] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3d0bc85),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.207] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.207] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:11.210] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:11.210] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:11.263] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:11.263] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:11.263] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:11.263] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:11.263] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:11.263] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:11.263] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:11.264] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:11.265] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.265] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@785bda13),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.265] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.266] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:11.266] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.266] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:11.268] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:11.269] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:11.285] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:11.285] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:11.285] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:11.285] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:11.286] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:11.286] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:11.286] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:11.287] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.287] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@21a79071),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.288] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.288] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:11.288] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.288] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:11.290] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:11.291] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.303] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:11.303] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:11.303] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.303] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:11.303] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:11.304] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:11.304] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:11.304] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.304] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:11.305] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:11.305] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:11.303] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7d08c1b7),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.303] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.303] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.303] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@208e2fb5),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.304] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.304] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@ffa490e),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.304] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:11.305] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:11.305] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:11.402] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:11.402] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:11.403] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.403] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:11.403] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.403] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:11.404] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:11.405] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.406] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@35e5ebbf),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.406] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.406] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:11.407] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@dda2737),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.407] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/second-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:11.410] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:11.411] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:11.461] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:11.461] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:11.461] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:11.461] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:11.461] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:11.461] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:11.462] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:11.462] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:11.464] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.464] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@427b2d29),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.464] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.464] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:11.465] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.465] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:11.468] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:11.468] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:11.483] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:11.483] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:11.484] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:11.484] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:11.484] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:11.484] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:11.484] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:11.485] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.486] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:11.486] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:11.486] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@44c8422),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.486] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.486] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:11.486] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.486] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[fourth] [DEBUG] [06/14/2012 08:19:11.486] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:11.488] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.488] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@77a6686),CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.488] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.488] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[second] [DEBUG] [06/14/2012 08:19:11.488] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.488] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3377, cap=3377)
[first] [DEBUG] [06/14/2012 08:19:11.489] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:11.490] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.492] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:11.492] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:11.561] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:11.561] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:11.561] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.561] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:11.561] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:11.561] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@24d200d8),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.561] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61378-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.561] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.561] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@774b9b80),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.562] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:11.562] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x56f052d5, /127.0.0.1:61378 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:11.562] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7b9bd735,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@3e4ac866),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.562] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:11.562] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:11.562] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4ddf3d59),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.562] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:11.563] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:11.563] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:11.563] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:11.563] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:11.582] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:11.582] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:11.582] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x11082823, /127.0.0.1:61382 => localhost/127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.582] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:11.582] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:11.582] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5554ea79,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@74e0eb3f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:11.582] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.583] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d6721bd, /127.0.0.1:61384 => localhost/127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.583] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@b74cb21,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@23137792),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:11.583] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:11.583] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/second-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.583] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@52352d87,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@2140243b),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.583] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:11.583] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61382-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:11.583] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@63220fd1,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@513c952f),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:11.583] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61384-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:11.584] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:11.584] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.584] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:11.584] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.588] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-3)
[second] [DEBUG] [06/14/2012 08:19:11.588] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-3)),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$m]
[second] [DEBUG] [06/14/2012 08:19:11.588] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:11.588] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61372: EnterBarrier(after-3)
[first] [DEBUG] [06/14/2012 08:19:11.589] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(EnterBarrier(after-3),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:11.589] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-3)
[first] [DEBUG] [06/14/2012 08:19:11.589] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-3),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-3,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]
[first] [DEBUG] [06/14/2012 08:19:11.589] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-3,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1])))
[fourth] [DEBUG] [06/14/2012 08:19:11.590] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-3)
[fourth] [DEBUG] [06/14/2012 08:19:11.590] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-3)),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$k]
[fourth] [DEBUG] [06/14/2012 08:19:11.590] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:11.590] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61377: EnterBarrier(after-3)
[first] [DEBUG] [06/14/2012 08:19:11.591] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(EnterBarrier(after-3),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:11.591] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-3)
[first] [DEBUG] [06/14/2012 08:19:11.591] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-3),Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-3,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4]
[first] [DEBUG] [06/14/2012 08:19:11.591] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1]), NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4])),after-3,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1])))
[first] [DEBUG] [06/14/2012 08:19:11.591] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/barriers] canceling timer 'Timeout'
[first] [DEBUG] [06/14/2012 08:19:11.592] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] processing Event(ToClient(BarrierResult(after-3,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:11.592] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] processing Event(ToClient(BarrierResult(after-3,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:11.592] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] processing Event(ToClient(BarrierResult(after-3,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:11.592] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/barriers] transition Waiting -> Idle
[first] [DEBUG] [06/14/2012 08:19:11.592] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: BarrierResult(after-3,true)
[fourth] [DEBUG] [06/14/2012 08:19:11.592] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-3,true)
[second] [DEBUG] [06/14/2012 08:19:11.592] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-3,true)
[first] [DEBUG] [06/14/2012 08:19:11.592] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-3,true),Data(Some([id: 0x34cef6f1, /127.0.0.1:61368 => /127.0.0.1:4711]),Some((after-3,Actor[akka://ConvergenceSpec/temp/$m])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:11.592] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-3
[second] [DEBUG] [06/14/2012 08:19:11.592] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-3,true),Data(Some([id: 0x52158cc6, /127.0.0.1:61372 => localhost/127.0.0.1:4711]),Some((after-3,Actor[akka://ConvergenceSpec/temp/$m])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:11.592] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-3
[fourth] [DEBUG] [06/14/2012 08:19:11.592] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-3,true),Data(Some([id: 0x53ea0105, /127.0.0.1:61377 => localhost/127.0.0.1:4711]),Some((after-3,Actor[akka://ConvergenceSpec/temp/$k])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:11.592] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-3
[fourth] - must not move a new joining node to Up while there is no convergence
[first] - must not move a new joining node to Up while there is no convergence
[second] - must not move a new joining node to Up while there is no convergence
[first] [DEBUG] [06/14/2012 08:19:11.594] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] disconnect from /127.0.0.1:61372
[second] [DEBUG] [06/14/2012 08:19:11.594] [ConvergenceSpec-akka.actor.default-dispatcher-1] [PlayerHandler(akka://ConvergenceSpec)] disconnected from localhost/127.0.0.1:4711
[first] [DEBUG] [06/14/2012 08:19:11.594] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] disconnect from /127.0.0.1:61368
[first] [DEBUG] [06/14/2012 08:19:11.594] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ConductorHandler(akka://ConvergenceSpec)] disconnect from /127.0.0.1:61377
[first] [DEBUG] [06/14/2012 08:19:11.595] [ConvergenceSpec-akka.actor.default-dispatcher-6] [PlayerHandler(akka://ConvergenceSpec)] disconnected from /127.0.0.1:4711
[fourth] [DEBUG] [06/14/2012 08:19:11.595] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] disconnected from localhost/127.0.0.1:4711
[second] [DEBUG] [06/14/2012 08:19:11.595] [ConvergenceSpec-akka.actor.default-dispatcher-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 :> localhost/127.0.0.1:4711] unbound
[second] [DEBUG] [06/14/2012 08:19:11.595] [ConvergenceSpec-akka.actor.default-dispatcher-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x52158cc6, /127.0.0.1:61372 :> localhost/127.0.0.1:4711] closed
[fourth] [DEBUG] [06/14/2012 08:19:11.596] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 :> localhost/127.0.0.1:4711] unbound
[first] [WARN] [06/14/2012 08:19:11.595] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61377-server4] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: ClientDisconnected
[first] [WARN] [06/14/2012 08:19:11.596] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61368-server1] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: ClientDisconnected
[first] [WARN] [06/14/2012 08:19:11.596] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61372-server2] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: ClientDisconnected
[fourth] [DEBUG] [06/14/2012 08:19:11.596] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x53ea0105, /127.0.0.1:61377 :> localhost/127.0.0.1:4711] closed
[fourth] [WARN] [06/14/2012 08:19:11.598] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: PoisonPill
[fourth] [DEBUG] [06/14/2012 08:19:11.600] [ConvergenceSpec-akka.actor.default-dispatcher-2] [EventStream] shutting down: StandardOutLogger started
[fourth] [DEBUG] [06/14/2012 08:19:11.602] [ConvergenceSpec-akka.actor.default-dispatcher-2] [EventStream] all default loggers stopped
[fourth] [INFO] [06/14/2012 08:19:11.606] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Shutting down cluster Node and cluster daemons...
[second] [WARN] [06/14/2012 08:19:11.596] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: PoisonPill
[second] [DEBUG] [06/14/2012 08:19:11.598] [ConvergenceSpec-akka.actor.default-dispatcher-1] [EventStream] shutting down: StandardOutLogger started
[second] [DEBUG] [06/14/2012 08:19:11.598] [ConvergenceSpec-akka.actor.default-dispatcher-1] [EventStream] shutting down: StandardOutLogger started
[second] [DEBUG] [06/14/2012 08:19:11.599] [ConvergenceSpec-akka.actor.default-dispatcher-1] [EventStream] all default loggers stopped
[second] [DEBUG] [06/14/2012 08:19:11.599] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:11.600] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:11.600] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.600] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:11.600] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:11.600] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:11.596] [ConvergenceSpec-akka.actor.default-dispatcher-6] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 :> /127.0.0.1:4711] unbound
[second] [DEBUG] [06/14/2012 08:19:11.601] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@second -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d), akka://ConvergenceSpec@first -> VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(295ff6d8eb4bbbbbdd8e91c6d37098d2) -> 00000137e9a522e9, Node(ca7a4d592be8916b235be61f6db2b361) -> 00000137e9a52cab, Node(4bed8358e1b2789b4f40e825137fe78e) -> 00000137e9a52d9d, Node(d15cb23c14bd3a0f6a00a3199f9d9708) -> 00000137e9a5237d))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:11.602] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x40395aaf, localhost/127.0.0.1:61384 => /127.0.0.1:61367] WRITE: CompositeChannelBuffer(ridx=0, widx=3377, cap=3377, components=2)
[second] [INFO] [06/14/2012 08:19:11.606] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Shutting down cluster Node and cluster daemons...
[first] [DEBUG] [06/14/2012 08:19:11.596] [ConvergenceSpec-akka.actor.default-dispatcher-6] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x34cef6f1, /127.0.0.1:61368 :> /127.0.0.1:4711] closed
[first] [WARN] [06/14/2012 08:19:11.596] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: PoisonPill
[first] [DEBUG] [06/14/2012 08:19:11.598] [ConvergenceSpec-akka.actor.default-dispatcher-3] [EventStream] shutting down: StandardOutLogger started
[first] [DEBUG] [06/14/2012 08:19:11.598] [ConvergenceSpec-akka.actor.default-dispatcher-3] [EventStream] shutting down: StandardOutLogger started
[first] [DEBUG] [06/14/2012 08:19:11.599] [ConvergenceSpec-akka.actor.default-dispatcher-3] [EventStream] all default loggers stopped
[first] [INFO] [06/14/2012 08:19:11.604] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Shutting down cluster Node and cluster daemons...
[first] [DEBUG] [06/14/2012 08:19:11.609] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [PassiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:11.610] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:11.610] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [PassiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:11.610] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [PassiveRemoteClient@akka://ConvergenceSpec@fourth]
[first] [INFO] [06/14/2012 08:19:11.610] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@fourth
[first] [DEBUG] [06/14/2012 08:19:11.610] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [PassiveRemoteClient@akka://ConvergenceSpec@fourth] has been shut down
[first] [DEBUG] [06/14/2012 08:19:11.610] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4398ee2f, localhost/127.0.0.1:61382 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[first] [DEBUG] [06/14/2012 08:19:11.610] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74e8f8c5, localhost/127.0.0.1:61378 => /127.0.0.1:61364] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:11.612] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:11.613] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@first]
[second] [INFO] [06/14/2012 08:19:11.613] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@first
[fourth] [INFO] [06/14/2012 08:19:11.613] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[second] [DEBUG] [06/14/2012 08:19:11.613] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x56f052d5, /127.0.0.1:61378 :> localhost/127.0.0.1:61364]
[second] [DEBUG] [06/14/2012 08:19:11.614] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x40395aaf, localhost/127.0.0.1:61384 :> /127.0.0.1:61367]
[first] [DEBUG] [06/14/2012 08:19:11.614] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x74e8f8c5, localhost/127.0.0.1:61378 :> /127.0.0.1:61364]
[second] [DEBUG] [06/14/2012 08:19:11.614] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@first] has been shut down
[second] [DEBUG] [06/14/2012 08:19:11.614] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [PassiveRemoteClient@akka://ConvergenceSpec@fourth]
[second] [INFO] [06/14/2012 08:19:11.614] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@fourth
[second] [DEBUG] [06/14/2012 08:19:11.614] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [PassiveRemoteClient@akka://ConvergenceSpec@fourth] has been shut down
[first] [DEBUG] [06/14/2012 08:19:11.614] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteServerClientClosed@akka://ConvergenceSpec@first: Client[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:11.614] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteServerClientClosed@akka://ConvergenceSpec@second: Client[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:11.615] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x3d6721bd, /127.0.0.1:61384 :> localhost/127.0.0.1:61367]
[fourth] [DEBUG] [06/14/2012 08:19:11.615] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:11.615] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@first]
[fourth] [INFO] [06/14/2012 08:19:11.616] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@first
[fourth] [DEBUG] [06/14/2012 08:19:11.616] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x11082823, /127.0.0.1:61382 :> localhost/127.0.0.1:61364]
[fourth] [DEBUG] [06/14/2012 08:19:11.616] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@first] has been shut down
[second] [INFO] [06/14/2012 08:19:11.618] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteServerShutdown@akka://ConvergenceSpec@second
[fourth] [INFO] [06/14/2012 08:19:11.620] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteServerShutdown@akka://ConvergenceSpec@fourth
[success] Total time: 12 s, completed Jun 14, 2012 8:19:11 AM
akka-cluster > multi-jvm:test-only akka.cluster.ConvergenceWithFailureDetectorPuppet
[info] * akka.cluster.ConvergenceWithFailureDetectorPuppet
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.156] [main] [EventStream(akka://ConvergenceSpec)] logger log1-TestEventListener started
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.158] [main] [EventStream(akka://ConvergenceSpec)] Default Loggers started
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.239] [main] [EventStream(akka://ConvergenceSpec)] logger log1-TestEventListener started
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.242] [main] [EventStream(akka://ConvergenceSpec)] Default Loggers started
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.340] [main] [EventStream(akka://ConvergenceSpec)] logger log1-TestEventListener started
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.343] [main] [EventStream(akka://ConvergenceSpec)] Default Loggers started
[JVM-Node1] [INFO] [06/14/2012 08:19:15.380] [main] [ActorSystem(ConvergenceSpec)] RemoteServerStarted@akka://ConvergenceSpec@localhost:61416
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.456] [main] [EventStream(akka://ConvergenceSpec)] logger log1-TestEventListener started
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.461] [main] [EventStream(akka://ConvergenceSpec)] Default Loggers started
[JVM-Node2] [INFO] [06/14/2012 08:19:15.479] [main] [ActorSystem(ConvergenceSpec)] RemoteServerStarted@akka://ConvergenceSpec@localhost:61419
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.515] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message GetSockAddr
[JVM-Node3] [INFO] [06/14/2012 08:19:15.560] [main] [ActorSystem(ConvergenceSpec)] RemoteServerStarted@akka://ConvergenceSpec@localhost:61422
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.597] [ConvergenceSpec-akka.actor.default-dispatcher-3] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b] open
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.631] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] connection from /127.0.0.1:61423
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.632] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] bound
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.633] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] connected to /127.0.0.1:4711
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.633] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message CreateServerFSM([id: 0x2a63b2e6, /127.0.0.1:61423 => /127.0.0.1:4711])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.645] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Connected([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Data(None,None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.665] [ConvergenceSpec-akka.actor.default-dispatcher-2] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c] open
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.668] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] connection from /127.0.0.1:61424
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.668] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message CreateServerFSM([id: 0x066de609, /127.0.0.1:61424 => /127.0.0.1:4711])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.671] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] transition Connecting -> AwaitDone
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.674] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 53
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.675] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] bound
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.676] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] connected to localhost/127.0.0.1:4711
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.695] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Connected([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Data(None,None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node4] [INFO] [06/14/2012 08:19:15.704] [main] [ActorSystem(ConvergenceSpec)] RemoteServerStarted@akka://ConvergenceSpec@localhost:61427
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.722] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 54
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.722] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] transition Connecting -> AwaitDone
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.743] [ConvergenceSpec-akka.actor.default-dispatcher-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x219fdbcb] open
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.745] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] connection from /127.0.0.1:61428
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.746] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message CreateServerFSM([id: 0x2a8a1977, /127.0.0.1:61428 => /127.0.0.1:4711])
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.751] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711] bound
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.752] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] connected to localhost/127.0.0.1:4711
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.773] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Connected([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),Data(None,None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.781] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: Hello(first,akka://ConvergenceSpec@localhost:61416)
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.782] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: Hello(second,akka://ConvergenceSpec@localhost:61419)
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.782] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(Hello(first,akka://ConvergenceSpec@localhost:61416),None) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.782] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(Hello(second,akka://ConvergenceSpec@localhost:61419),None) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.782] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] transition Initial -> Ready
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.783] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] transition Initial -> Ready
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.783] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message NodeInfo(RoleName(second),akka://ConvergenceSpec@localhost:61419,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.783] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message NodeInfo(RoleName(first),akka://ConvergenceSpec@localhost:61416,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.784] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/barriers] processing Event(NodeInfo(RoleName(second),akka://ConvergenceSpec@localhost:61419,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]),Data(Set(),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.785] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/barriers] processing Event(NodeInfo(RoleName(first),akka://ConvergenceSpec@localhost:61416,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@localhost:61419,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.793] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] transition Connecting -> AwaitDone
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.794] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61428: Hello(third,akka://ConvergenceSpec@localhost:61422)
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.795] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(Hello(third,akka://ConvergenceSpec@localhost:61422),None) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.795] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] transition Initial -> Ready
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.795] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message NodeInfo(RoleName(third),akka://ConvergenceSpec@localhost:61422,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.795] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/barriers] processing Event(NodeInfo(RoleName(third),akka://ConvergenceSpec@localhost:61422,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@localhost:61419,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@localhost:61416,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.795] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711] written 53
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.841] [ConvergenceSpec-akka.actor.default-dispatcher-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624] open
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.843] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] connection from /127.0.0.1:61429
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.843] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message CreateServerFSM([id: 0x18f63055, /127.0.0.1:61429 => /127.0.0.1:4711])
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.848] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] bound
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.848] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] connected to localhost/127.0.0.1:4711
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.859] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Connected([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),Data(None,None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.875] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] transition Connecting -> AwaitDone
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.876] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61429: Hello(fourth,akka://ConvergenceSpec@localhost:61427)
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.876] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(Hello(fourth,akka://ConvergenceSpec@localhost:61427),None) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.876] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] written 54
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.876] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] transition Initial -> Ready
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.877] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message NodeInfo(RoleName(fourth),akka://ConvergenceSpec@localhost:61427,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.877] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(NodeInfo(RoleName(fourth),akka://ConvergenceSpec@localhost:61427,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@localhost:61419,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@localhost:61416,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@localhost:61422,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.880] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(Done),None) from Actor[akka://ConvergenceSpec/user/controller]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.880] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(Done),None) from Actor[akka://ConvergenceSpec/user/controller]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.880] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(ToClient(Done),None) from Actor[akka://ConvergenceSpec/user/controller]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.881] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(ToClient(Done),None) from Actor[akka://ConvergenceSpec/user/controller]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.883] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: Done
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.883] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Done,Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.883] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] received Done: starting test
[JVM-Node1] [DEBUG] [06/14/2012 08:19:15.883] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] transition AwaitDone -> Connected
[JVM-Node1] [INFO] [06/14/2012 08:19:15.887] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode1(akka://ConvergenceSpec)] Role [first] started
[first] ConvergenceWithFailureDetectorPuppetMultiJvmNode1:
[first] A cluster of 3 members 
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.960] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: Done
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.960] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Done,Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.960] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] received Done: starting test
[JVM-Node3] [DEBUG] [06/14/2012 08:19:15.960] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] transition AwaitDone -> Connected
[JVM-Node3] [INFO] [06/14/2012 08:19:15.965] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode3(akka://ConvergenceSpec)] Role [third] started
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.976] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: Done
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.977] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Done,Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.977] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] received Done: starting test
[JVM-Node2] [DEBUG] [06/14/2012 08:19:15.977] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] transition AwaitDone -> Connected
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.981] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: Done
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.982] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(Done,Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.982] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] received Done: starting test
[JVM-Node4] [DEBUG] [06/14/2012 08:19:15.982] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] transition AwaitDone -> Connected
[JVM-Node4] [INFO] [06/14/2012 08:19:15.987] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode4(akka://ConvergenceSpec)] Role [fourth] started
[JVM-Node2] [INFO] [06/14/2012 08:19:15.985] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode2(akka://ConvergenceSpec)] Role [second] started
[third] ConvergenceWithFailureDetectorPuppetMultiJvmNode3:
[third] A cluster of 3 members 
[third] [DEBUG] [06/14/2012 08:19:16.029] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-started)
[second] ConvergenceWithFailureDetectorPuppetMultiJvmNode2:
[third] [DEBUG] [06/14/2012 08:19:16.031] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-started)),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$a]
[third] [DEBUG] [06/14/2012 08:19:16.032] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711] written 21
[first] [DEBUG] [06/14/2012 08:19:16.033] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61428: EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:16.033] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(EnterBarrier(first-started),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:16.033] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:16.034] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-started),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]
[first] [DEBUG] [06/14/2012 08:19:16.035] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] setting timer 'Timeout'/30000 milliseconds: StateTimeout
[first] [DEBUG] [06/14/2012 08:19:16.035] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] transition Idle -> Waiting
[second] A cluster of 3 members 
[second] [DEBUG] [06/14/2012 08:19:16.041] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-started)
[fourth] ConvergenceWithFailureDetectorPuppetMultiJvmNode4:
[second] [DEBUG] [06/14/2012 08:19:16.044] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-started)),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$a]
[second] [DEBUG] [06/14/2012 08:19:16.045] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 21
[first] [DEBUG] [06/14/2012 08:19:16.045] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:16.045] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(EnterBarrier(first-started),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:16.045] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:16.046] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-started),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]
[first] [DEBUG] [06/14/2012 08:19:16.047] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3])))
[fourth] A cluster of 3 members 
[fourth] [DEBUG] [06/14/2012 08:19:16.054] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-started)
[fourth] [DEBUG] [06/14/2012 08:19:16.056] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-started)),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$a]
[fourth] [DEBUG] [06/14/2012 08:19:16.057] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] written 21
[first] [DEBUG] [06/14/2012 08:19:16.057] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61429: EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:16.058] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(EnterBarrier(first-started),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:16.058] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:16.058] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-started),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]
[first] [DEBUG] [06/14/2012 08:19:16.059] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3])))
[first] [INFO] [06/14/2012 08:19:16.060] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - is starting up...
[first] [INFO] [06/14/2012 08:19:16.080] [main] [Node(akka://ConvergenceSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[first] [INFO] [06/14/2012 08:19:16.095] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - registering cluster JMX MBean [akka:type=Cluster]
[first] [INFO] [06/14/2012 08:19:16.096] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - has started up successfully
[first] [DEBUG] [06/14/2012 08:19:16.096] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-started)
[first] [DEBUG] [06/14/2012 08:19:16.098] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-started)),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$a]
[first] [DEBUG] [06/14/2012 08:19:16.099] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 21
[first] [DEBUG] [06/14/2012 08:19:16.099] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:16.099] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(EnterBarrier(first-started),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:16.099] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-started)
[first] [DEBUG] [06/14/2012 08:19:16.100] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-started),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]
[first] [DEBUG] [06/14/2012 08:19:16.100] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-started,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3])))
[first] [DEBUG] [06/14/2012 08:19:16.101] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] canceling timer 'Timeout'
[first] [DEBUG] [06/14/2012 08:19:16.101] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(BarrierResult(first-started,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:16.101] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(ToClient(BarrierResult(first-started,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:16.102] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(ToClient(BarrierResult(first-started,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:16.102] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(BarrierResult(first-started,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:16.102] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/barriers] transition Waiting -> Idle
[first] [DEBUG] [06/14/2012 08:19:16.102] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: BarrierResult(first-started,true)
[first] [DEBUG] [06/14/2012 08:19:16.102] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-started,true),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Some((first-started,Actor[akka://ConvergenceSpec/temp/$a])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:16.102] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-started,true)
[fourth] [DEBUG] [06/14/2012 08:19:16.102] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-started,true),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),Some((first-started,Actor[akka://ConvergenceSpec/temp/$a])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:16.102] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-started,true)
[second] [DEBUG] [06/14/2012 08:19:16.102] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-started,true),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((first-started,Actor[akka://ConvergenceSpec/temp/$a])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:16.102] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-started,true)
[third] [DEBUG] [06/14/2012 08:19:16.102] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-started,true),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),Some((first-started,Actor[akka://ConvergenceSpec/temp/$a])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:16.103] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-started
[third] [DEBUG] [06/14/2012 08:19:16.103] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-started
[second] [DEBUG] [06/14/2012 08:19:16.103] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-started
[fourth] [DEBUG] [06/14/2012 08:19:16.103] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-started
[fourth] [DEBUG] [06/14/2012 08:19:16.104] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-second-third-joined)
[fourth] [DEBUG] [06/14/2012 08:19:16.104] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-second-third-joined)),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$b]
[fourth] [DEBUG] [06/14/2012 08:19:16.105] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] written 33
[first] [DEBUG] [06/14/2012 08:19:16.105] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61429: EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:16.105] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(EnterBarrier(first-second-third-joined),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:16.106] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:16.106] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-second-third-joined),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]
[first] [DEBUG] [06/14/2012 08:19:16.106] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] setting timer 'Timeout'/30000 milliseconds: StateTimeout
[first] [DEBUG] [06/14/2012 08:19:16.106] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] transition Idle -> Waiting
[second] [INFO] [06/14/2012 08:19:16.193] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - is starting up...
[third] [INFO] [06/14/2012 08:19:16.195] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - is starting up...
[second] [INFO] [06/14/2012 08:19:16.216] [main] [Node(akka://ConvergenceSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[third] [INFO] [06/14/2012 08:19:16.222] [main] [Node(akka://ConvergenceSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[second] [INFO] [06/14/2012 08:19:16.231] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - registering cluster JMX MBean [akka:type=Cluster]
[second] [INFO] [06/14/2012 08:19:16.232] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - has started up successfully
[second] [DEBUG] [06/14/2012 08:19:16.233] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$b]
[second] [DEBUG] [06/14/2012 08:19:16.234] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:16.235] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:16.235] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:16.237] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[first] [DEBUG] [06/14/2012 08:19:16.235] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:16.235] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[second] [DEBUG] [06/14/2012 08:19:16.237] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$b])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [INFO] [06/14/2012 08:19:16.238] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Trying to send JOIN to [akka://ConvergenceSpec@first] through connection [Actor[akka://ConvergenceSpec@first/system/cluster/commands]]
[third] [INFO] [06/14/2012 08:19:16.242] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - registering cluster JMX MBean [akka:type=Cluster]
[third] [INFO] [06/14/2012 08:19:16.243] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - has started up successfully
[second] [DEBUG] [06/14/2012 08:19:16.243] [main] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@first|localhost/127.0.0.1]
[third] [DEBUG] [06/14/2012 08:19:16.244] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$b]
[third] [DEBUG] [06/14/2012 08:19:16.246] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:16.246] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61428: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:16.246] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:16.246] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:16.246] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[third] [DEBUG] [06/14/2012 08:19:16.247] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[third] [DEBUG] [06/14/2012 08:19:16.248] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$b])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [INFO] [06/14/2012 08:19:16.249] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Trying to send JOIN to [akka://ConvergenceSpec@first] through connection [Actor[akka://ConvergenceSpec@first/system/cluster/commands]]
[third] [DEBUG] [06/14/2012 08:19:16.253] [main] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@first|localhost/127.0.0.1]
[second] [DEBUG] [06/14/2012 08:19:16.268] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@first
[second] [DEBUG] [06/14/2012 08:19:16.272] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[second] [INFO] [06/14/2012 08:19:16.274] [main] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@first
[second] [DEBUG] [06/14/2012 08:19:16.275] [main] [RemoteClient(akka://ConvergenceSpec)] Sending message Join(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/commands]
[third] [DEBUG] [06/14/2012 08:19:16.278] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@first
[second] [DEBUG] [06/14/2012 08:19:16.280] [main] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.ClusterAction$Join]
[third] [DEBUG] [06/14/2012 08:19:16.281] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[third] [INFO] [06/14/2012 08:19:16.283] [main] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@first
[third] [DEBUG] [06/14/2012 08:19:16.283] [main] [RemoteClient(akka://ConvergenceSpec)] Sending message Join(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@first/system/cluster/commands]
[third] [DEBUG] [06/14/2012 08:19:16.289] [main] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.ClusterAction$Join]
[second] [DEBUG] [06/14/2012 08:19:16.295] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6d3136e5),CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.295] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.296] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@localhost:61430
[first] [DEBUG] [06/14/2012 08:19:16.296] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@localhost:61431
[first] [DEBUG] [06/14/2012 08:19:16.297] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[second] [DEBUG] [06/14/2012 08:19:16.299] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.299] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@58d51a54),CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.300] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.300] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[third] [DEBUG] [06/14/2012 08:19:16.304] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4fb595f3),CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.304] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.305] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[third] [DEBUG] [06/14/2012 08:19:16.309] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[third] [DEBUG] [06/14/2012 08:19:16.309] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@74e8f8c5),CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.309] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.310] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[first] [DEBUG] [06/14/2012 08:19:16.320] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.320] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.320] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[first] [DEBUG] [06/14/2012 08:19:16.320] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[first] [DEBUG] [06/14/2012 08:19:16.322] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.322] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[first] [DEBUG] [06/14/2012 08:19:16.322] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.322] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[first] [INFO] [06/14/2012 08:19:16.325] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:16.325] [ConvergenceSpec-2] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:16.325] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:16.326] [ConvergenceSpec-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:16.326] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteServerClientConnected@akka://ConvergenceSpec@first: Client[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:16.327] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteServerClientConnected@akka://ConvergenceSpec@first: Client[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:16.340] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Join(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/commands]<+{akka://ConvergenceSpec@first/system/cluster/commands} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:16.341] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Join(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/commands]<+{akka://ConvergenceSpec@first/system/cluster/commands} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [INFO] [06/14/2012 08:19:16.341] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Node [akka://ConvergenceSpec@third] is JOINING
[first] [DEBUG] [06/14/2012 08:19:16.343] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:16.345] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Node [akka://ConvergenceSpec@second] is JOINING
[first] [DEBUG] [06/14/2012 08:19:16.346] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:16.415] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:16.417] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:16.418] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:16.418] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:16.418] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:16.419] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to members not in seen table [Set(akka://ConvergenceSpec@second, akka://ConvergenceSpec@third)].
[first] [DEBUG] [06/14/2012 08:19:16.420] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:16.421] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:16.426] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.GossipEnvelope]
[first] [DEBUG] [06/14/2012 08:19:16.451] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2518, cap=2518, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.451] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6e75d758),CompositeChannelBuffer(ridx=0, widx=2518, cap=2518, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.451] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2518, cap=2518, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.452] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:16.452] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1494, cap=1494)
[second] [DEBUG] [06/14/2012 08:19:16.452] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.452] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:16.453] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=1494, cap=1494)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.454] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1494, cap=1494)
[second] [DEBUG] [06/14/2012 08:19:16.485] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [INFO] [06/14/2012 08:19:16.493] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Can't establish a causal relationship between "remote" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa))] and "local" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a55777)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@second, status = Joining)], meta = [], version = VectorClock(Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a55777))] - merging them into [Gossip(overview = GossipOverview(seen = [], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c))]
[second] [DEBUG] [06/14/2012 08:19:16.494] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:16.515] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:16.515] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:16.515] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.Heartbeat]
[first] [DEBUG] [06/14/2012 08:19:16.516] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.516] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:16.516] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:16.516] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@10320399),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.517] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.517] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:16.517] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.517] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@27ed688f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.517] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.517] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.517] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:16.517] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:16.518] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@239cd5f5),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.518] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:16.519] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:16.519] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:16.525] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:16.527] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:16.551] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:16.553] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:16.554] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:16.554] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:16.554] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:16.558] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:16.555] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:16.560] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:16.560] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.GossipEnvelope]
[third] [DEBUG] [06/14/2012 08:19:16.563] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@third, status = Joining)]
[third] [INFO] [06/14/2012 08:19:16.563] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Leader is moving node [akka://ConvergenceSpec@third] from JOINING to UP
[second] [DEBUG] [06/14/2012 08:19:16.566] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.567] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@399197b),CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.567] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.567] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[first] [DEBUG] [06/14/2012 08:19:16.567] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1814, cap=1814)
[first] [DEBUG] [06/14/2012 08:19:16.567] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.568] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:16.568] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:16.568] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:16.568] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:16.569] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=1814, cap=1814)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.569] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@third
[second] [DEBUG] [06/14/2012 08:19:16.570] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[second] [INFO] [06/14/2012 08:19:16.570] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:16.569] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1814, cap=1814)
[second] [DEBUG] [06/14/2012 08:19:16.570] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:16.571] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@46fb6763),CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.571] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.572] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.572] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6e30fc1f),CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.573] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2838, cap=2838, components=2)
[third] [DEBUG] [06/14/2012 08:19:16.573] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@localhost:61432
[third] [DEBUG] [06/14/2012 08:19:16.573] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[third] [DEBUG] [06/14/2012 08:19:16.573] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1856, cap=1856)
[third] [DEBUG] [06/14/2012 08:19:16.577] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.577] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[first] [DEBUG] [06/14/2012 08:19:16.578] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:16.579] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=1856, cap=1856)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [INFO] [06/14/2012 08:19:16.579] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@second
[third] [DEBUG] [06/14/2012 08:19:16.579] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1856, cap=1856)
[third] [DEBUG] [06/14/2012 08:19:16.580] [ConvergenceSpec-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:16.581] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteServerClientConnected@akka://ConvergenceSpec@third: Client[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:16.582] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:16.613] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:16.613] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:16.613] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:16.614] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:16.614] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:16.614] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to members not in seen table [Set(akka://ConvergenceSpec@third)].
[first] [DEBUG] [06/14/2012 08:19:16.614] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:16.614] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:16.616] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.617] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5057f57f),CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.617] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2836, cap=2836, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.620] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:16.620] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=788, cap=788)
[second] [DEBUG] [06/14/2012 08:19:16.620] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.620] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:16.620] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=788, cap=788)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.621] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=788, cap=788)
[second] [DEBUG] [06/14/2012 08:19:16.625] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:16.625] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:16.626] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[third] [INFO] [06/14/2012 08:19:16.635] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Can't establish a causal relationship between "remote" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c))] and "local" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a558d4)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a558d4))] - merging them into [Gossip(overview = GossipOverview(seen = [], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))]
[third] [DEBUG] [06/14/2012 08:19:16.635] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:16.651] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:16.651] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:16.651] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.Heartbeat]
[second] [DEBUG] [06/14/2012 08:19:16.652] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.652] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:16.653] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:16.653] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:16.654] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.654] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:16.655] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:16.655] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:16.658] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:16.658] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:16.658] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.Heartbeat]
[third] [DEBUG] [06/14/2012 08:19:16.658] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:16.658] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:16.659] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:16.659] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6a643edd),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.659] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:16.659] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:16.659] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4959d87f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.659] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.652] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4a005364),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.652] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.653] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:16.653] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.653] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3c870e2f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.653] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.660] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:16.660] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@46b44eb4),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.660] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:16.661] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:16.661] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:16.653] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.654] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:16.655] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:16.655] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:16.659] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:16.659] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.659] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:16.660] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:16.661] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:16.749] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:16.749] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:16.749] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:16.750] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:16.750] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:16.750] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:16.750] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:16.752] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=2896, cap=2896, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.753] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@114562c5),CompositeChannelBuffer(ridx=0, widx=2896, cap=2896, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.753] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2896, cap=2896, components=2)
[third] [DEBUG] [06/14/2012 08:19:16.753] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[third] [DEBUG] [06/14/2012 08:19:16.754] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=848, cap=848)
[third] [DEBUG] [06/14/2012 08:19:16.754] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.754] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[third] [DEBUG] [06/14/2012 08:19:16.754] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=848, cap=848)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.754] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=848, cap=848)
[third] [DEBUG] [06/14/2012 08:19:16.755] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[third] [DEBUG] [06/14/2012 08:19:16.756] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:16.756] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:16.757] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:16.757] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:16.758] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:16.759] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:16.759] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:16.760] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:16.760] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.GossipEnvelope]
[third] [DEBUG] [06/14/2012 08:19:16.767] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)
[third] [DEBUG] [06/14/2012 08:19:16.767] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7691a4fb),CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.767] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.768] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[first] [DEBUG] [06/14/2012 08:19:16.768] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1876, cap=1876)
[first] [DEBUG] [06/14/2012 08:19:16.768] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.768] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[first] [DEBUG] [06/14/2012 08:19:16.768] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=1876, cap=1876)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.769] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1876, cap=1876)
[third] [DEBUG] [06/14/2012 08:19:16.768] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:16.769] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:16.770] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:16.771] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)
[third] [DEBUG] [06/14/2012 08:19:16.772] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@521ba1d9),CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.772] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2900, cap=2900, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.772] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:16.772] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1876, cap=1876)
[second] [DEBUG] [06/14/2012 08:19:16.773] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@46b44eb4),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.773] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:16.773] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@46b44eb4),BigEndianHeapChannelBuffer(ridx=0, widx=1876, cap=1876)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.773] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1876, cap=1876)
[first] [DEBUG] [06/14/2012 08:19:16.776] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:16.777] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:16.781] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:16.781] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:16.812] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:16.813] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:16.813] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:16.814] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:16.814] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:16.815] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to members not in seen table [Set(akka://ConvergenceSpec@second)].
[first] [DEBUG] [06/14/2012 08:19:16.815] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:16.815] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:16.817] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.818] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@193a83cc),CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.818] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)
[third] [DEBUG] [06/14/2012 08:19:16.818] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[third] [DEBUG] [06/14/2012 08:19:16.819] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1972, cap=1972)
[third] [DEBUG] [06/14/2012 08:19:16.819] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@239cd5f5),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.819] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[third] [DEBUG] [06/14/2012 08:19:16.819] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@239cd5f5),BigEndianHeapChannelBuffer(ridx=0, widx=1972, cap=1972)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.820] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1972, cap=1972)
[third] [DEBUG] [06/14/2012 08:19:16.826] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:16.826] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:16.910] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:16.911] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:16.911] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.911] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:16.912] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:16.912] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@50a2d304),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.912] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.912] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.912] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:16.913] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.912] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7d0ade7e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.912] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.913] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:16.913] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:16.913] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@239cd5f5),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.914] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:16.914] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:16.914] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:16.915] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:16.915] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:16.947] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:16.947] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:16.947] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:16.947] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:16.947] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:16.947] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:16.948] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:16.950] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.950] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@54956254),CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.951] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2996, cap=2996, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.962] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:16.962] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)
[second] [DEBUG] [06/14/2012 08:19:16.962] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@46b44eb4),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.962] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:16.963] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@46b44eb4),BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:16.963] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)
[first] [DEBUG] [06/14/2012 08:19:16.958] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:16.958] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)
[first] [DEBUG] [06/14/2012 08:19:16.959] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.959] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:16.959] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:16.959] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)
[third] [DEBUG] [06/14/2012 08:19:16.951] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2996, cap=2996)
[third] [DEBUG] [06/14/2012 08:19:16.951] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=2996, cap=2996)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.951] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2996, cap=2996)
[third] [DEBUG] [06/14/2012 08:19:16.953] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[third] [DEBUG] [06/14/2012 08:19:16.954] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:16.954] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:16.954] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:16.954] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:16.954] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:16.955] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:16.964] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:16.956] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:16.956] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:16.957] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[third] [DEBUG] [06/14/2012 08:19:16.958] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:16.958] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:16.958] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5e35dc95),CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.958] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[first] [DEBUG] [06/14/2012 08:19:16.964] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:16.959] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:16.961] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[third] [DEBUG] [06/14/2012 08:19:16.961] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3857dc15),CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:16.962] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[second] [DEBUG] [06/14/2012 08:19:16.967] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:16.967] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.008] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:17.009] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:17.009] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:17.009] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.009] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.009] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to members not in seen table [Set(akka://ConvergenceSpec@second)].
[first] [DEBUG] [06/14/2012 08:19:17.010] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:17.010] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:17.011] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.011] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@99be6ee),CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.011] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.012] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[third] [DEBUG] [06/14/2012 08:19:17.012] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)
[third] [DEBUG] [06/14/2012 08:19:17.012] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@239cd5f5),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.012] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[third] [DEBUG] [06/14/2012 08:19:17.012] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@239cd5f5),BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.013] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)
[third] [DEBUG] [06/14/2012 08:19:17.016] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.017] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:17.046] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:17.046] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.047] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.047] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:17.047] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.047] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1b6aa42e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.048] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.047] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:17.048] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.048] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:17.049] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.049] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:17.052] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.052] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:17.052] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.047] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.047] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.048] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1aa632be),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.048] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.053] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:17.052] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.053] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@10320399),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.053] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:17.053] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.053] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.053] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@27ed688f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.054] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.048] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.048] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.048] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.049] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:17.053] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.053] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.053] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:17.054] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@46b44eb4),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.054] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.054] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.054] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.055] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.055] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.145] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:17.145] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:17.146] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:17.146] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.146] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.146] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:17.146] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.148] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.148] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1915470e),CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.148] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.148] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:17.149] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)
[first] [DEBUG] [06/14/2012 08:19:17.149] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.149] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:17.149] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.149] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1008, cap=1008)
[third] [DEBUG] [06/14/2012 08:19:17.151] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[third] [DEBUG] [06/14/2012 08:19:17.151] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:17.152] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:17.152] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:17.152] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:17.152] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.152] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:17.153] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.154] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:17.154] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.154] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6699176f),CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.155] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3056, cap=3056, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.155] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3056, cap=3056)
[second] [DEBUG] [06/14/2012 08:19:17.155] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@46b44eb4),BigEndianHeapChannelBuffer(ridx=0, widx=3056, cap=3056)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.155] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3056, cap=3056)
[second] [DEBUG] [06/14/2012 08:19:17.159] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.160] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.207] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:17.207] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:17.208] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:17.208] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.208] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.208] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:17.208] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)]
[first] [INFO] [06/14/2012 08:19:17.209] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Leader is moving node [akka://ConvergenceSpec@first] from JOINING to UP
[first] [DEBUG] [06/14/2012 08:19:17.209] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [INFO] [06/14/2012 08:19:17.209] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Leader is moving node [akka://ConvergenceSpec@second] from JOINING to UP
[first] [DEBUG] [06/14/2012 08:19:17.210] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.210] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2d9845fb),CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.210] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.211] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)
[third] [DEBUG] [06/14/2012 08:19:17.211] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@239cd5f5),BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.211] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)
[third] [DEBUG] [06/14/2012 08:19:17.214] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.214] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[third] [DEBUG] [06/14/2012 08:19:17.214] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:17.306] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:17.306] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:17.307] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.307] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:17.307] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2206179e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.307] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.307] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.307] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.307] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.308] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.308] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.308] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@ba3bc8c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.308] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.308] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.308] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@239cd5f5),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.308] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:17.309] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.309] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:17.309] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.309] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:17.314] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:17.343] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:17.343] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:17.343] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:17.343] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.343] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.344] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.349] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)
[third] [DEBUG] [06/14/2012 08:19:17.350] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:17.346] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)
[third] [DEBUG] [06/14/2012 08:19:17.350] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.350] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)
[third] [DEBUG] [06/14/2012 08:19:17.350] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:17.350] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:17.344] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.345] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.351] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:17.351] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:17.351] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.352] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:17.346] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.347] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2968, cap=2968)
[first] [DEBUG] [06/14/2012 08:19:17.350] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.351] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:17.346] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.346] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@465da37c),CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.346] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.346] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:17.347] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.349] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.349] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2206179e),CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.349] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2968, cap=2968, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.353] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.354] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7548c02f),CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.354] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.354] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.354] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[first] [DEBUG] [06/14/2012 08:19:17.354] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.355] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:17.355] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[first] [DEBUG] [06/14/2012 08:19:17.358] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.358] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.405] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:17.405] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:17.405] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:17.405] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.405] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.406] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[first] [DEBUG] [06/14/2012 08:19:17.406] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:17.406] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:17.407] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2866, cap=2866, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.407] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@32eccfe7),CompositeChannelBuffer(ridx=0, widx=2866, cap=2866, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.407] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2866, cap=2866, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.408] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:17.408] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=818, cap=818)
[second] [DEBUG] [06/14/2012 08:19:17.408] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.408] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:17.408] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=818, cap=818)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.409] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=818, cap=818)
[second] [DEBUG] [06/14/2012 08:19:17.411] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.412] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:17.415] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:17.442] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:17.442] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.443] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.443] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:17.443] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.443] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3df89785),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.444] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.443] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:17.444] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.444] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:17.445] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.445] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:17.448] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.448] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:17.448] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.448] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.449] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:17.449] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7b4189d0),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.449] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.449] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.449] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@63f796d0),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.450] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.443] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.443] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.444] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2aaf914c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.444] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.450] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:17.450] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@46b44eb4),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.450] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:17.451] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.451] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.444] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.444] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.445] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.445] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:17.449] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.449] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.449] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.450] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.451] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.511] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[first] [DEBUG] [06/14/2012 08:19:17.515] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:17.541] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:17.541] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:17.541] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:17.541] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.541] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.542] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:17.542] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.543] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.543] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.544] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:17.544] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@49e4585),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.544] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.544] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[first] [DEBUG] [06/14/2012 08:19:17.544] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.544] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[second] [DEBUG] [06/14/2012 08:19:17.544] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.546] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.546] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@566e9e64),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.547] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.546] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[first] [DEBUG] [06/14/2012 08:19:17.548] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:17.546] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.551] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[second] [DEBUG] [06/14/2012 08:19:17.551] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@46b44eb4),BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.551] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3028, cap=3028)
[third] [DEBUG] [06/14/2012 08:19:17.546] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.547] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[third] [DEBUG] [06/14/2012 08:19:17.547] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[third] [DEBUG] [06/14/2012 08:19:17.548] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:17.548] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:17.548] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:17.554] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.554] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:17.548] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:17.548] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.549] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Joining), Member(address = akka://ConvergenceSpec@second, status = Joining), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:17.550] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.550] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@384ab40a),CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.550] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.550] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3028, cap=3028, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.551] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:17.603] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:17.603] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:17.604] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:17.604] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.604] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.604] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[first] [DEBUG] [06/14/2012 08:19:17.604] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:17.604] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:17.605] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.606] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@13ef45e0),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.606] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.606] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[third] [DEBUG] [06/14/2012 08:19:17.606] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@239cd5f5),BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.607] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[third] [DEBUG] [06/14/2012 08:19:17.609] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.610] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:17.611] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[first] [DEBUG] [06/14/2012 08:19:17.615] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[third] [DEBUG] [06/14/2012 08:19:17.618] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)]
[third] [DEBUG] [06/14/2012 08:19:17.619] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-second-third-joined)
[third] [DEBUG] [06/14/2012 08:19:17.619] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-second-third-joined)),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$c]
[third] [DEBUG] [06/14/2012 08:19:17.620] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711] written 33
[first] [DEBUG] [06/14/2012 08:19:17.620] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61428: EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:17.620] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(EnterBarrier(first-second-third-joined),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.620] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:17.621] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-second-third-joined),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]
[first] [DEBUG] [06/14/2012 08:19:17.621] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])))
[first] [DEBUG] [06/14/2012 08:19:17.702] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:17.703] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:17.703] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.704] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:17.704] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:17.704] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@41d47f59),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.705] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.705] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.705] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.705] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2af4ca49),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.705] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.705] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.706] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:17.706] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:17.706] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@239cd5f5),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.707] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:17.707] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.708] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:17.708] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.709] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:17.711] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[first] [DEBUG] [06/14/2012 08:19:17.716] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, since not all nodes have seen the same state yet. [2 of 3]
[second] [DEBUG] [06/14/2012 08:19:17.738] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:17.739] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:17.739] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:17.739] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.739] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.740] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:17.750] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[third] [DEBUG] [06/14/2012 08:19:17.744] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[first] [DEBUG] [06/14/2012 08:19:17.750] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.750] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[second] [DEBUG] [06/14/2012 08:19:17.741] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.743] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.743] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2773a64a),CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.743] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2926, cap=2926, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.744] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.744] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2926, cap=2926)
[third] [DEBUG] [06/14/2012 08:19:17.745] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Initiating new round of gossip
[third] [DEBUG] [06/14/2012 08:19:17.746] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[third] [DEBUG] [06/14/2012 08:19:17.746] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:17.746] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:17.746] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[third] [DEBUG] [06/14/2012 08:19:17.746] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.747] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:17.748] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a557fa, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.748] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Receiving gossip from [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:17.749] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.749] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:17.749] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6c6c2a78),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.750] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.750] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.751] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:17.753] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.753] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4838462a),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.753] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.753] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[second] [DEBUG] [06/14/2012 08:19:17.754] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@46b44eb4),BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.754] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[first] [DEBUG] [06/14/2012 08:19:17.754] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.755] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.757] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@third,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.757] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.801] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:17.801] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:17.802] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:17.802] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.802] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.802] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:17.802] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)]
[first] [DEBUG] [06/14/2012 08:19:17.803] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:17.804] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.804] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6d8f729c),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.804] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.805] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[second] [DEBUG] [06/14/2012 08:19:17.805] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.805] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[second] [DEBUG] [06/14/2012 08:19:17.808] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.809] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:17.812] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)]
[second] [DEBUG] [06/14/2012 08:19:17.813] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-second-third-joined)
[second] [DEBUG] [06/14/2012 08:19:17.814] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-second-third-joined)),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$c]
[second] [DEBUG] [06/14/2012 08:19:17.814] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 33
[first] [DEBUG] [06/14/2012 08:19:17.814] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:17.815] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(EnterBarrier(first-second-third-joined),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.815] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:17.815] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-second-third-joined),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]
[first] [DEBUG] [06/14/2012 08:19:17.816] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])))
[first] [DEBUG] [06/14/2012 08:19:17.816] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)]
[first] [DEBUG] [06/14/2012 08:19:17.817] [main] [ActorSystem(ConvergenceSpec)] entering barriers (first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:17.818] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(first-second-third-joined)),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$b]
[first] [DEBUG] [06/14/2012 08:19:17.818] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 33
[first] [DEBUG] [06/14/2012 08:19:17.819] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:17.819] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(EnterBarrier(first-second-third-joined),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.819] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(first-second-third-joined)
[first] [DEBUG] [06/14/2012 08:19:17.820] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(first-second-third-joined),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]
[first] [DEBUG] [06/14/2012 08:19:17.820] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),first-second-third-joined,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])))
[first] [DEBUG] [06/14/2012 08:19:17.820] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] canceling timer 'Timeout'
[first] [DEBUG] [06/14/2012 08:19:17.820] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(BarrierResult(first-second-third-joined,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.820] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(BarrierResult(first-second-third-joined,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.820] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(ToClient(BarrierResult(first-second-third-joined,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.821] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/barriers] transition Waiting -> Idle
[first] [DEBUG] [06/14/2012 08:19:17.821] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(ToClient(BarrierResult(first-second-third-joined,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[second] [DEBUG] [06/14/2012 08:19:17.821] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-second-third-joined,true)
[first] [DEBUG] [06/14/2012 08:19:17.821] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: BarrierResult(first-second-third-joined,true)
[second] [DEBUG] [06/14/2012 08:19:17.821] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-second-third-joined,true),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((first-second-third-joined,Actor[akka://ConvergenceSpec/temp/$c])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.821] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-second-third-joined,true),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Some((first-second-third-joined,Actor[akka://ConvergenceSpec/temp/$b])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.821] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-second-third-joined
[fourth] [DEBUG] [06/14/2012 08:19:17.821] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-second-third-joined,true)
[third] [DEBUG] [06/14/2012 08:19:17.821] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(first-second-third-joined,true)
[third] [DEBUG] [06/14/2012 08:19:17.821] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-second-third-joined,true),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),Some((first-second-third-joined,Actor[akka://ConvergenceSpec/temp/$c])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.821] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-second-third-joined
[second] [DEBUG] [06/14/2012 08:19:17.821] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-second-third-joined
[fourth] [DEBUG] [06/14/2012 08:19:17.822] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(first-second-third-joined,true),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),Some((first-second-third-joined,Actor[akka://ConvergenceSpec/temp/$b])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:17.822] [main] [ActorSystem(ConvergenceSpec)] passed barrier first-second-third-joined
[third] [DEBUG] [06/14/2012 08:19:17.822] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-1)
[second] [DEBUG] [06/14/2012 08:19:17.822] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-1)
[first] [DEBUG] [06/14/2012 08:19:17.822] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-1)
[second] [DEBUG] [06/14/2012 08:19:17.822] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-1)),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$d]
[third] [DEBUG] [06/14/2012 08:19:17.822] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-1)),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$d]
[first] [DEBUG] [06/14/2012 08:19:17.822] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-1)),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$c]
[first] [DEBUG] [06/14/2012 08:19:17.823] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 15
[third] [DEBUG] [06/14/2012 08:19:17.823] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:17.823] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: EnterBarrier(after-1)
[fourth] [DEBUG] [06/14/2012 08:19:17.823] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-1)
[fourth] [DEBUG] [06/14/2012 08:19:17.823] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-1)),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$c]
[first] [DEBUG] [06/14/2012 08:19:17.823] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: EnterBarrier(after-1)
[second] [DEBUG] [06/14/2012 08:19:17.823] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:17.823] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(EnterBarrier(after-1),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.823] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(EnterBarrier(after-1),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.823] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-1)
[fourth] [DEBUG] [06/14/2012 08:19:17.823] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:17.824] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:17.824] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61428: EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:17.824] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(EnterBarrier(after-1),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.824] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:17.824] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61429: EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:17.824] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-1),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]
[first] [DEBUG] [06/14/2012 08:19:17.824] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(EnterBarrier(after-1),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.825] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-1)
[first] [DEBUG] [06/14/2012 08:19:17.825] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] setting timer 'Timeout'/30000 milliseconds: StateTimeout
[first] [DEBUG] [06/14/2012 08:19:17.825] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] transition Idle -> Waiting
[first] [DEBUG] [06/14/2012 08:19:17.825] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-1),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]
[first] [DEBUG] [06/14/2012 08:19:17.825] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1])))
[first] [DEBUG] [06/14/2012 08:19:17.825] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-1),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]
[first] [DEBUG] [06/14/2012 08:19:17.826] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1])))
[first] [DEBUG] [06/14/2012 08:19:17.826] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-1),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]
[first] [DEBUG] [06/14/2012 08:19:17.827] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),after-1,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1])))
[first] [DEBUG] [06/14/2012 08:19:17.827] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] canceling timer 'Timeout'
[first] [DEBUG] [06/14/2012 08:19:17.827] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(ToClient(BarrierResult(after-1,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.827] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(ToClient(BarrierResult(after-1,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.827] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] transition Waiting -> Idle
[first] [DEBUG] [06/14/2012 08:19:17.828] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(BarrierResult(after-1,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.828] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(BarrierResult(after-1,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.828] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: BarrierResult(after-1,true)
[third] [DEBUG] [06/14/2012 08:19:17.828] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-1,true)
[second] [DEBUG] [06/14/2012 08:19:17.828] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-1,true)
[second] [DEBUG] [06/14/2012 08:19:17.828] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-1,true),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((after-1,Actor[akka://ConvergenceSpec/temp/$d])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.828] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-1
[fourth] [DEBUG] [06/14/2012 08:19:17.828] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-1,true)
[fourth] [DEBUG] [06/14/2012 08:19:17.828] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-1,true),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),Some((after-1,Actor[akka://ConvergenceSpec/temp/$c])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.828] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-1,true),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),Some((after-1,Actor[akka://ConvergenceSpec/temp/$d])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:17.829] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-1
[first] [DEBUG] [06/14/2012 08:19:17.828] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-1,true),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Some((after-1,Actor[akka://ConvergenceSpec/temp/$c])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.828] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-1
[third] [DEBUG] [06/14/2012 08:19:17.829] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-1
[first] - must reach initial convergence
[second] - must reach initial convergence
[first] [DEBUG] [06/14/2012 08:19:17.831] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(third))),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$d]
[third] - must reach initial convergence
[second] [DEBUG] [06/14/2012 08:19:17.832] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(third))),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$e]
[fourth] - must reach initial convergence
[third] [DEBUG] [06/14/2012 08:19:17.832] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(third))),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$e]
[first] [DEBUG] [06/14/2012 08:19:17.832] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:17.832] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: GetAddress(RoleName(third))
[third] [DEBUG] [06/14/2012 08:19:17.832] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711] written 13
[second] [DEBUG] [06/14/2012 08:19:17.832] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:17.833] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:17.833] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(GetAddress(RoleName(third)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.833] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:17.833] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61428: GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:17.833] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(GetAddress(RoleName(third)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.833] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(GetAddress(RoleName(third)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:17.833] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(third))),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$d]
[first] [DEBUG] [06/14/2012 08:19:17.834] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:17.834] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(third))
[second] [DEBUG] [06/14/2012 08:19:17.834] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(third),akka://ConvergenceSpec@third)
[second] [DEBUG] [06/14/2012 08:19:17.834] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(third),akka://ConvergenceSpec@third),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((third,Actor[akka://ConvergenceSpec/temp/$e])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.834] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(third),akka://ConvergenceSpec@third)
[first] [DEBUG] [06/14/2012 08:19:17.834] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(AddressReply(RoleName(third),akka://ConvergenceSpec@third)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:17.834] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(ToClient(AddressReply(RoleName(third),akka://ConvergenceSpec@third)),None) from Actor[akka://ConvergenceSpec/user/controller]
[third] [DEBUG] [06/14/2012 08:19:17.834] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(third),akka://ConvergenceSpec@third),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),Some((third,Actor[akka://ConvergenceSpec/temp/$e])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.834] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(AddressReply(RoleName(third),akka://ConvergenceSpec@third)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:17.834] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(third),akka://ConvergenceSpec@third)
[first] [DEBUG] [06/14/2012 08:19:17.835] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(third),akka://ConvergenceSpec@third),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Some((third,Actor[akka://ConvergenceSpec/temp/$d])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.835] [main] [ActorSystem(ConvergenceSpec)] entering barriers (before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:17.835] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(before-shutdown)),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$e]
[second] [DEBUG] [06/14/2012 08:19:17.835] [main] [ActorSystem(ConvergenceSpec)] entering barriers (before-shutdown)
[third] [DEBUG] [06/14/2012 08:19:17.835] [main] [ActorSystem(ConvergenceSpec)] entering barriers (before-shutdown)
[second] [DEBUG] [06/14/2012 08:19:17.835] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(before-shutdown)),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$f]
[second] [DEBUG] [06/14/2012 08:19:17.836] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 23
[second] [DEBUG] [06/14/2012 08:19:17.838] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:17.838] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.838] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.839] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:17.839] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@28f549de),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.839] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.839] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.839] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.839] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@469e0c2d),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.839] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.842] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(before-shutdown,true)
[second] [DEBUG] [06/14/2012 08:19:17.842] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(before-shutdown,true),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((before-shutdown,Actor[akka://ConvergenceSpec/temp/$f])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.842] [main] [ActorSystem(ConvergenceSpec)] passed barrier before-shutdown
[second] [DEBUG] [06/14/2012 08:19:17.844] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$g]
[second] [DEBUG] [06/14/2012 08:19:17.844] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 13
[second] [DEBUG] [06/14/2012 08:19:17.846] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:17.846] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[second] [DEBUG] [06/14/2012 08:19:17.846] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@46b44eb4),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.847] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/third-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:17.847] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$g])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.847] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(second))),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$h]
[second] [DEBUG] [06/14/2012 08:19:17.848] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.848] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 14
[second] [DEBUG] [06/14/2012 08:19:17.848] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.849] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(second),akka://ConvergenceSpec@second)
[second] [DEBUG] [06/14/2012 08:19:17.849] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(second),akka://ConvergenceSpec@second),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((second,Actor[akka://ConvergenceSpec/temp/$h])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.835] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 23
[first] [DEBUG] [06/14/2012 08:19:17.836] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:17.836] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61428: EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:17.836] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61429: GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:17.836] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:17.836] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(EnterBarrier(before-shutdown),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.836] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(EnterBarrier(before-shutdown),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.837] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(EnterBarrier(before-shutdown),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.837] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:17.837] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(GetAddress(RoleName(third)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.837] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:17.837] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:17.837] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(third))
[first] [DEBUG] [06/14/2012 08:19:17.837] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(ToClient(AddressReply(RoleName(third),akka://ConvergenceSpec@third)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:17.837] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(before-shutdown),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]
[first] [DEBUG] [06/14/2012 08:19:17.837] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] setting timer 'Timeout'/30000 milliseconds: StateTimeout
[first] [DEBUG] [06/14/2012 08:19:17.837] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] transition Idle -> Waiting
[first] [DEBUG] [06/14/2012 08:19:17.838] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(before-shutdown),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]
[first] [DEBUG] [06/14/2012 08:19:17.838] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3])))
[first] [DEBUG] [06/14/2012 08:19:17.838] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(before-shutdown),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]
[first] [DEBUG] [06/14/2012 08:19:17.838] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3])))
[first] [DEBUG] [06/14/2012 08:19:17.839] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:17.835] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:17.839] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.840] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61429: EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:17.840] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.840] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(EnterBarrier(before-shutdown),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.840] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(before-shutdown)
[first] [DEBUG] [06/14/2012 08:19:17.840] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(before-shutdown),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]
[first] [DEBUG] [06/14/2012 08:19:17.840] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.840] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:17.835] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(before-shutdown)),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$f]
[third] [DEBUG] [06/14/2012 08:19:17.836] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711] written 23
[third] [DEBUG] [06/14/2012 08:19:17.840] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:17.840] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@11b9d4d8),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.840] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[third] [DEBUG] [06/14/2012 08:19:17.841] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@third)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@third/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.841] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[third] [DEBUG] [06/14/2012 08:19:17.841] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(before-shutdown,true)
[third] [DEBUG] [06/14/2012 08:19:17.842] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(before-shutdown,true),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),Some((before-shutdown,Actor[akka://ConvergenceSpec/temp/$f])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.842] [main] [ActorSystem(ConvergenceSpec)] passed barrier before-shutdown
[third] [DEBUG] [06/14/2012 08:19:17.843] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-2)
[third] [DEBUG] [06/14/2012 08:19:17.843] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-2)),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$g]
[third] [DEBUG] [06/14/2012 08:19:17.844] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711] written 15
[third] [DEBUG] [06/14/2012 08:19:17.844] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.844] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:17.845] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x715c2843, /127.0.0.1:61431 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.845] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@third] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[third] [DEBUG] [06/14/2012 08:19:17.845] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@third) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[third] [DEBUG] [06/14/2012 08:19:17.845] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@135d4273,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6964130),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.845] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.845] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x344e17f9, localhost/127.0.0.1:61432 => /127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.846] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5f9299f5,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3d360c93),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[third] [DEBUG] [06/14/2012 08:19:17.846] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61432-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[third] [DEBUG] [06/14/2012 08:19:17.846] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: TerminateMsg(0)
[third] [DEBUG] [06/14/2012 08:19:17.847] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(TerminateMsg(0),Data(Some([id: 0x219fdbcb, /127.0.0.1:61428 => localhost/127.0.0.1:4711]),Some((after-2,Actor[akka://ConvergenceSpec/temp/$g])))) from Actor[akka://ConvergenceSpec/deadLetters]
[third] [DEBUG] [06/14/2012 08:19:17.847] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-2,false)
[first] [DEBUG] [06/14/2012 08:19:17.841] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),before-shutdown,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3])))
[first] [DEBUG] [06/14/2012 08:19:17.841] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] canceling timer 'Timeout'
[first] [DEBUG] [06/14/2012 08:19:17.841] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(ToClient(BarrierResult(before-shutdown,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.841] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(BarrierResult(before-shutdown,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.841] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/barriers] transition Waiting -> Idle
[first] [DEBUG] [06/14/2012 08:19:17.841] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(BarrierResult(before-shutdown,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.842] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(ToClient(BarrierResult(before-shutdown,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.842] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: BarrierResult(before-shutdown,true)
[first] [DEBUG] [06/14/2012 08:19:17.842] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(before-shutdown,true),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Some((before-shutdown,Actor[akka://ConvergenceSpec/temp/$e])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.842] [main] [ActorSystem(ConvergenceSpec)] passed barrier before-shutdown
[first] [DEBUG] [06/14/2012 08:19:17.843] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message Terminate(RoleName(third),0)
[first] [DEBUG] [06/14/2012 08:19:17.844] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61428: EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:17.844] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(EnterBarrier(after-2),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.844] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:17.844] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61429: EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:17.844] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.845] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(EnterBarrier(after-2),None) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:17.838] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(third),akka://ConvergenceSpec@third)
[fourth] [DEBUG] [06/14/2012 08:19:17.838] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(third),akka://ConvergenceSpec@third),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),Some((third,Actor[akka://ConvergenceSpec/temp/$d])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:17.839] [main] [ActorSystem(ConvergenceSpec)] entering barriers (before-shutdown)
[fourth] [DEBUG] [06/14/2012 08:19:17.839] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(before-shutdown)),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$e]
[fourth] [DEBUG] [06/14/2012 08:19:17.839] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] written 23
[fourth] [DEBUG] [06/14/2012 08:19:17.842] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(before-shutdown,true)
[fourth] [DEBUG] [06/14/2012 08:19:17.842] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(before-shutdown,true),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),Some((before-shutdown,Actor[akka://ConvergenceSpec/temp/$e])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:17.842] [main] [ActorSystem(ConvergenceSpec)] passed barrier before-shutdown
[fourth] [DEBUG] [06/14/2012 08:19:17.843] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-2)
[fourth] [DEBUG] [06/14/2012 08:19:17.844] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-2)),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$f]
[fourth] [DEBUG] [06/14/2012 08:19:17.844] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:17.845] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:17.845] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:17.845] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:17.846] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(ToClient(TerminateMsg(0)),None) from Actor[akka://ConvergenceSpec/temp/$f]
[first] [DEBUG] [06/14/2012 08:19:17.846] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(RemoveClient(RoleName(third)),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(third),akka://ConvergenceSpec@third,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:17.851] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.852] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$g]
[first] [DEBUG] [06/14/2012 08:19:17.852] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:17.852] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:17.852] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:17.852] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:17.852] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:17.852] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.852] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:17.852] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-2),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3]
[first] [DEBUG] [06/14/2012 08:19:17.852] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:17.853] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(ToClient(BarrierResult(after-2,false)),Some(Actor[akka://ConvergenceSpec/temp/$f])) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:17.853] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@third) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.853] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-2),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),,List())) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]
[first] [DEBUG] [06/14/2012 08:19:17.853] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] setting timer 'Timeout'/30000 milliseconds: StateTimeout
[first] [DEBUG] [06/14/2012 08:19:17.853] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:17.853] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/barriers] transition Idle -> Waiting
[first] [DEBUG] [06/14/2012 08:19:17.853] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[first] [DEBUG] [06/14/2012 08:19:17.853] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$g])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.853] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(second))),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$h]
[first] [DEBUG] [06/14/2012 08:19:17.853] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:17.854] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 14
[first] [DEBUG] [06/14/2012 08:19:17.854] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(GetAddress(RoleName(second)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.854] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:17.854] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:17.854] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(AddressReply(RoleName(second),akka://ConvergenceSpec@second)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:17.854] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(GetAddress(RoleName(second)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:17.854] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:17.854] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(AddressReply(RoleName(second),akka://ConvergenceSpec@second)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:17.854] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(second),akka://ConvergenceSpec@second)
[first] [DEBUG] [06/14/2012 08:19:17.854] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(second),akka://ConvergenceSpec@second),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Some((second,Actor[akka://ConvergenceSpec/temp/$h])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:17.937] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:17.937] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:17.937] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:17.938] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.938] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:17.938] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:17.939] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:17.942] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[second] [DEBUG] [06/14/2012 08:19:17.942] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@73f5eb7f),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:17.942] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[first] [DEBUG] [06/14/2012 08:19:17.999] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:17.999] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:18.000] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:18.000] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:18.000] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS available [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:18.000] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.000] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)]
[first] [INFO] [06/14/2012 08:19:18.000] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Marking node(s) as UNREACHABLE [Member(address = akka://ConvergenceSpec@third, status = Up)]
[first] [DEBUG] [06/14/2012 08:19:18.001] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:18.002] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x76136c55, localhost/127.0.0.1:61431 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.002] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5c232697),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.002] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61431-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.098] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.099] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:18.099] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.100] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@c58f769),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.100] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.101] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:18.101] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.101] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:18.103] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.103] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:18.135] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:18.136] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:18.136] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:18.136] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:18.137] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:18.137] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.141] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[second] [DEBUG] [06/14/2012 08:19:18.138] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:18.140] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.140] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:18.140] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:18.141] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@95d0a50),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.141] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.141] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.142] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2875, cap=2875)
[first] [DEBUG] [06/14/2012 08:19:18.146] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.146] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:18.142] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@third, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:18.143] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 => localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.144] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@590eb535),CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.144] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/third-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=2875, cap=2875, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.175] [New I/O client worker #1-2] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x438a3a1d, /127.0.0.1:61432 :> localhost/127.0.0.1:61422]
[first] [DEBUG] [06/14/2012 08:19:18.175] [New I/O server worker #1-2] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x76136c55, localhost/127.0.0.1:61431 :> /127.0.0.1:61416]
[first] [DEBUG] [06/14/2012 08:19:18.175] [New I/O server worker #2-3] [ConductorHandler(akka://ConvergenceSpec)] disconnect from /127.0.0.1:61428
[first] [DEBUG] [06/14/2012 08:19:18.176] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61428-server3] processing Event(ClientDisconnected,Some(Actor[akka://ConvergenceSpec/temp/$f])) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.177] [ConvergenceSpec-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [PassiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:18.179] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:18.179] [ConvergenceSpec-4] [RemoteClient(akka://ConvergenceSpec)] [PassiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:18.181] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteServerClientClosed@akka://ConvergenceSpec@first: Client[akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:18.196] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:18.197] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:18.197] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:18.197] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:18.197] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.197] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:18.198] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:18.200] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.201] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4aa193fe),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.201] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.201] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [DEBUG] [06/14/2012 08:19:18.201] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.201] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [DEBUG] [06/14/2012 08:19:18.207] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.207] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:18.234] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:18.234] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:18.234] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.234] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@63a0eec3),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.235] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:18.234] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.235] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.235] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:18.236] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.236] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:18.259] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[second] [DEBUG] [06/14/2012 08:19:18.260] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-2)
[second] [DEBUG] [06/14/2012 08:19:18.260] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-2)),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$i]
[second] [DEBUG] [06/14/2012 08:19:18.261] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:18.261] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:18.261] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(EnterBarrier(after-2),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.261] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:18.261] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-2),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),after-2,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]
[first] [DEBUG] [06/14/2012 08:19:18.262] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),after-2,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])))
[second] [DEBUG] [06/14/2012 08:19:18.333] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:18.333] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:18.334] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:18.334] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:18.334] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:18.336] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:18.339] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.339] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3bfc47),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.340] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.340] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:18.340] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.340] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[first] [DEBUG] [06/14/2012 08:19:18.340] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.341] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [DEBUG] [06/14/2012 08:19:18.341] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:18.343] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x438a3a1d, /127.0.0.1:61432 :> localhost/127.0.0.1:61422] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.345] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.345] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:18.346] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@7c9b87f6,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2583d260),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.346] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/third-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [WARN] [06/14/2012 08:19:18.349] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec@third/system/cluster/gossip] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)))
[second] [DEBUG] [06/14/2012 08:19:18.349] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [ERROR] [06/14/2012 08:19:18.350] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.nio.channels.ClosedChannelException]
[second] [INFO] [06/14/2012 08:19:18.350] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:18.353] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:18.359] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:18.360] [main] [ActorSystem(ConvergenceSpec)] entering barriers (after-2)
[first] [DEBUG] [06/14/2012 08:19:18.361] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(EnterBarrier(after-2)),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$i]
[first] [DEBUG] [06/14/2012 08:19:18.361] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 15
[first] [DEBUG] [06/14/2012 08:19:18.361] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:18.361] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(EnterBarrier(after-2),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.361] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message EnterBarrier(after-2)
[first] [DEBUG] [06/14/2012 08:19:18.362] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller/barriers] processing Event(EnterBarrier(after-2),Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),after-2,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4]))) from Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]
[first] [DEBUG] [06/14/2012 08:19:18.362] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller/barriers] handleBarrier(Data(Set(NodeInfo(RoleName(second),akka://ConvergenceSpec@second,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2]), NodeInfo(RoleName(first),akka://ConvergenceSpec@first,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1]), NodeInfo(RoleName(fourth),akka://ConvergenceSpec@fourth,Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])),after-2,List(Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2], Actor[akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4])))
[first] [DEBUG] [06/14/2012 08:19:18.362] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller/barriers] canceling timer 'Timeout'
[first] [DEBUG] [06/14/2012 08:19:18.362] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller/barriers] transition Waiting -> Idle
[first] [DEBUG] [06/14/2012 08:19:18.362] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(BarrierResult(after-2,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:18.362] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(BarrierResult(after-2,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:18.362] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(ToClient(BarrierResult(after-2,true)),None) from Actor[akka://ConvergenceSpec/user/controller/barriers]
[first] [DEBUG] [06/14/2012 08:19:18.362] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: BarrierResult(after-2,true)
[second] [DEBUG] [06/14/2012 08:19:18.363] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-2,true)
[fourth] [DEBUG] [06/14/2012 08:19:18.363] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: BarrierResult(after-2,true)
[first] [DEBUG] [06/14/2012 08:19:18.363] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-2,true),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Some((after-2,Actor[akka://ConvergenceSpec/temp/$i])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.363] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-2
[first] - must not reach convergence while any nodes are unreachable
[second] [DEBUG] [06/14/2012 08:19:18.363] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-2,true),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((after-2,Actor[akka://ConvergenceSpec/temp/$i])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.363] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-2
[fourth] [DEBUG] [06/14/2012 08:19:18.363] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(BarrierResult(after-2,true),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),Some((after-2,Actor[akka://ConvergenceSpec/temp/$f])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] - must not reach convergence while any nodes are unreachable
[fourth] [DEBUG] [06/14/2012 08:19:18.363] [main] [ActorSystem(ConvergenceSpec)] passed barrier after-2
[fourth] - must not reach convergence while any nodes are unreachable
[first] [DEBUG] [06/14/2012 08:19:18.364] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$j]
[first] [DEBUG] [06/14/2012 08:19:18.364] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:18.365] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: GetAddress(RoleName(first))
[second] [DEBUG] [06/14/2012 08:19:18.365] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$j]
[second] [DEBUG] [06/14/2012 08:19:18.365] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:18.365] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.365] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:18.365] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:18.365] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:18.365] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.365] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:18.366] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:18.366] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[first] [DEBUG] [06/14/2012 08:19:18.366] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$j])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.366] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[second] [DEBUG] [06/14/2012 08:19:18.366] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$j])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.367] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(second))),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$k]
[first] [DEBUG] [06/14/2012 08:19:18.367] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 14
[second] [DEBUG] [06/14/2012 08:19:18.367] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(second))),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$k]
[first] [DEBUG] [06/14/2012 08:19:18.367] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:18.367] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(GetAddress(RoleName(second)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.367] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(second))
[second] [DEBUG] [06/14/2012 08:19:18.367] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 14
[first] [DEBUG] [06/14/2012 08:19:18.367] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(AddressReply(RoleName(second),akka://ConvergenceSpec@second)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:18.368] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: GetAddress(RoleName(second))
[second] [DEBUG] [06/14/2012 08:19:18.368] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(second),akka://ConvergenceSpec@second)
[first] [DEBUG] [06/14/2012 08:19:18.368] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(GetAddress(RoleName(second)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.368] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:18.368] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(AddressReply(RoleName(second),akka://ConvergenceSpec@second)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:18.368] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(second),akka://ConvergenceSpec@second)
[first] [DEBUG] [06/14/2012 08:19:18.368] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(second),akka://ConvergenceSpec@second),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Some((second,Actor[akka://ConvergenceSpec/temp/$k])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.368] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(fourth))),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$l]
[first] [DEBUG] [06/14/2012 08:19:18.369] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711] written 14
[first] [DEBUG] [06/14/2012 08:19:18.369] [New I/O server worker #2-1] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61423: GetAddress(RoleName(fourth))
[first] [DEBUG] [06/14/2012 08:19:18.369] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(GetAddress(RoleName(fourth)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.369] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(fourth))
[first] [DEBUG] [06/14/2012 08:19:18.369] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] processing Event(ToClient(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:18.370] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61424: GetAddress(RoleName(fourth))
[first] [DEBUG] [06/14/2012 08:19:18.370] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from /127.0.0.1:4711: AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)
[first] [DEBUG] [06/14/2012 08:19:18.370] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(GetAddress(RoleName(fourth)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.370] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(fourth))
[first] [DEBUG] [06/14/2012 08:19:18.370] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth),Data(Some([id: 0x0a1e940b, /127.0.0.1:61423 => /127.0.0.1:4711]),Some((fourth,Actor[akka://ConvergenceSpec/temp/$l])))) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.370] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ToClient(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:18.371] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode1(akka://ConvergenceSpec)] assertNotMovedUp#1
[second] [DEBUG] [06/14/2012 08:19:18.369] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(second),akka://ConvergenceSpec@second),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((second,Actor[akka://ConvergenceSpec/temp/$k])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.369] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(fourth))),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$l]
[second] [DEBUG] [06/14/2012 08:19:18.370] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711] written 14
[second] [DEBUG] [06/14/2012 08:19:18.371] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)
[second] [DEBUG] [06/14/2012 08:19:18.371] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth),Data(Some([id: 0x0f786a3c, /127.0.0.1:61424 => localhost/127.0.0.1:4711]),Some((fourth,Actor[akka://ConvergenceSpec/temp/$l])))) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.372] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode2(akka://ConvergenceSpec)] assertNotMovedUp#1
[first] [DEBUG] [06/14/2012 08:19:18.395] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:18.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:18.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:18.396] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:18.396] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:18.396] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.397] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:18.398] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.399] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@70bb6255),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.399] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.399] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [DEBUG] [06/14/2012 08:19:18.399] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.400] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[second] [DEBUG] [06/14/2012 08:19:18.403] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.404] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [INFO] [06/14/2012 08:19:18.451] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - is starting up...
[fourth] [INFO] [06/14/2012 08:19:18.471] [main] [Node(akka://ConvergenceSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[fourth] [INFO] [06/14/2012 08:19:18.485] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - registering cluster JMX MBean [akka:type=Cluster]
[fourth] [INFO] [06/14/2012 08:19:18.486] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - has started up successfully
[fourth] [DEBUG] [06/14/2012 08:19:18.486] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$g]
[fourth] [DEBUG] [06/14/2012 08:19:18.487] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] written 13
[first] [DEBUG] [06/14/2012 08:19:18.487] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61429: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:18.487] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.495] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:18.488] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[second] [DEBUG] [06/14/2012 08:19:18.495] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.487] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:18.487] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:18.494] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.494] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:18.494] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.495] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@74d8c3ee),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.495] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.495] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:18.496] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.497] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:18.488] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$g])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [INFO] [06/14/2012 08:19:18.489] [main] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Trying to send JOIN to [akka://ConvergenceSpec@first] through connection [Actor[akka://ConvergenceSpec@first/system/cluster/commands]]
[fourth] [DEBUG] [06/14/2012 08:19:18.493] [main] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@first|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:18.516] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@localhost:61433
[fourth] [DEBUG] [06/14/2012 08:19:18.517] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@first
[fourth] [DEBUG] [06/14/2012 08:19:18.520] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[fourth] [INFO] [06/14/2012 08:19:18.521] [main] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@first
[fourth] [DEBUG] [06/14/2012 08:19:18.522] [main] [RemoteClient(akka://ConvergenceSpec)] Sending message Join(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/commands]
[fourth] [DEBUG] [06/14/2012 08:19:18.527] [main] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.ClusterAction$Join]
[second] [DEBUG] [06/14/2012 08:19:18.531] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:18.531] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:18.531] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:18.531] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:18.531] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:18.545] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4914a947),CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:18.546] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.536] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[first] [DEBUG] [06/14/2012 08:19:18.537] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.537] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3020, cap=3020)
[first] [DEBUG] [06/14/2012 08:19:18.541] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.541] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:18.549] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[first] [DEBUG] [06/14/2012 08:19:18.550] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.550] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61429: GetAddress(RoleName(first))
[first] [DEBUG] [06/14/2012 08:19:18.550] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[first] [DEBUG] [06/14/2012 08:19:18.550] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[first] [DEBUG] [06/14/2012 08:19:18.550] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(GetAddress(RoleName(first)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.551] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.551] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=482, cap=482)
[first] [DEBUG] [06/14/2012 08:19:18.551] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(first))
[first] [INFO] [06/14/2012 08:19:18.551] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@fourth
[first] [DEBUG] [06/14/2012 08:19:18.551] [ConvergenceSpec-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:18.551] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(ToClient(AddressReply(RoleName(first),akka://ConvergenceSpec@first)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:18.551] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteServerClientConnected@akka://ConvergenceSpec@first: Client[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:18.551] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Join(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/commands]<+{akka://ConvergenceSpec@first/system/cluster/commands} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [INFO] [06/14/2012 08:19:18.552] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Node [akka://ConvergenceSpec@fourth] is JOINING
[first] [DEBUG] [06/14/2012 08:19:18.552] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61429: GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:18.552] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:18.552] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(GetAddress(RoleName(second)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.552] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(second))
[first] [DEBUG] [06/14/2012 08:19:18.553] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(ToClient(AddressReply(RoleName(second),akka://ConvergenceSpec@second)),None) from Actor[akka://ConvergenceSpec/user/controller]
[first] [DEBUG] [06/14/2012 08:19:18.554] [New I/O server worker #2-4] [ConductorHandler(akka://ConvergenceSpec)] message from /127.0.0.1:61429: GetAddress(RoleName(fourth))
[first] [DEBUG] [06/14/2012 08:19:18.554] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(GetAddress(RoleName(fourth)),None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.554] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/user/controller] received handled message GetAddress(RoleName(fourth))
[fourth] [DEBUG] [06/14/2012 08:19:18.548] [main] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:18.549] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@647fc8b2),CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:18.549] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=482, cap=482, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:18.549] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(first))),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$h]
[fourth] [DEBUG] [06/14/2012 08:19:18.550] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] written 13
[fourth] [DEBUG] [06/14/2012 08:19:18.551] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(first),akka://ConvergenceSpec@first)
[fourth] [DEBUG] [06/14/2012 08:19:18.551] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(first),akka://ConvergenceSpec@first),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),Some((first,Actor[akka://ConvergenceSpec/temp/$h])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:18.552] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(second))),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$i]
[fourth] [DEBUG] [06/14/2012 08:19:18.552] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] written 14
[fourth] [DEBUG] [06/14/2012 08:19:18.553] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(second),akka://ConvergenceSpec@second)
[fourth] [DEBUG] [06/14/2012 08:19:18.553] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(second),akka://ConvergenceSpec@second),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),Some((second,Actor[akka://ConvergenceSpec/temp/$i])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:18.553] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(ToServer(GetAddress(RoleName(fourth))),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),None)) from Actor[akka://ConvergenceSpec/temp/$j]
[fourth] [DEBUG] [06/14/2012 08:19:18.554] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711] written 14
[fourth] [DEBUG] [06/14/2012 08:19:18.555] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] message from localhost/127.0.0.1:4711: AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)
[fourth] [DEBUG] [06/14/2012 08:19:18.555] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/user/TestConductorClient] processing Event(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth),Data(Some([id: 0x05c09624, /127.0.0.1:61429 => localhost/127.0.0.1:4711]),Some((fourth,Actor[akka://ConvergenceSpec/temp/$j])))) from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:18.556] [main] [ConvergenceWithFailureDetectorPuppetMultiJvmNode4(akka://ConvergenceSpec)] assertNotMovedUp#1
[second] [DEBUG] [06/14/2012 08:19:18.532] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:18.534] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.536] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[second] [DEBUG] [06/14/2012 08:19:18.536] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.554] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] processing Event(ToClient(AddressReply(RoleName(fourth),akka://ConvergenceSpec@fourth)),None) from Actor[akka://ConvergenceSpec/user/controller]
[second] [DEBUG] [06/14/2012 08:19:18.536] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2484de3c),CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.593] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:18.593] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:18.593] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:18.593] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:18.536] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.536] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[second] [ERROR] [06/14/2012 08:19:18.539] [ConvergenceSpec-akka.actor.default-dispatcher-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[second] [ERROR] [06/14/2012 08:19:18.541] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:18.593] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:18.543] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:18.545] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0f1a47df] WRITE: CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.546] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3020, cap=3020, components=2) in direction Send when socket [id: 0x0f1a47df] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:18.593] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[second] [DEBUG] [06/14/2012 08:19:18.550] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[second] [INFO] [06/14/2012 08:19:18.550] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[second] [DEBUG] [06/14/2012 08:19:18.550] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:18.594] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.600] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:18.605] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.606] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6063f5af),CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.606] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3215, cap=3215, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.606] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3215, cap=3215)
[second] [DEBUG] [06/14/2012 08:19:18.606] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=3215, cap=3215)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.606] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3215, cap=3215)
[second] [DEBUG] [06/14/2012 08:19:18.609] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55e70, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.610] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:18.630] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:18.630] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:18.630] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.631] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:18.632] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@fourth|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:18.636] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@localhost:61435
[first] [DEBUG] [06/14/2012 08:19:18.632] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:18.637] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x539c5048, localhost/127.0.0.1:61435 => /127.0.0.1:61427] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[fourth] [DEBUG] [06/14/2012 08:19:18.637] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x539c5048, localhost/127.0.0.1:61435 => /127.0.0.1:61427] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:18.632] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@49d29a78),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.632] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.634] [New I/O client worker #1-4] [FailureInjector(akka://ConvergenceSpec)] connected to akka://@fourth
[second] [DEBUG] [06/14/2012 08:19:18.635] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x219b6225, /127.0.0.1:61435 => localhost/127.0.0.1:61427] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[second] [INFO] [06/14/2012 08:19:18.635] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@fourth
[second] [DEBUG] [06/14/2012 08:19:18.635] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:18.635] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x219b6225, /127.0.0.1:61435 => localhost/127.0.0.1:61427] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.635] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/fourth-snd4] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@38f8ff42,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@633e6346),CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.636] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/fourth-snd4] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.637] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/fourth-snd4] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@38f8ff42,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@c9c6201),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.637] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/fourth-snd4] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.633] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.633] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:18.633] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.634] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:18.640] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@42bb0406,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4054824),BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:18.640] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=42, cap=42)
[fourth] [DEBUG] [06/14/2012 08:19:18.641] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@42bb0406,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4054824),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:18.641] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [INFO] [06/14/2012 08:19:18.641] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientStarted@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:18.641] [ConvergenceSpec-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:18.642] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteServerClientConnected@akka://ConvergenceSpec@fourth: Client[akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:18.649] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:18.651] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:18.729] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:18.730] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:18.730] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:18.730] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:18.730] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:18.730] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:18.731] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:18.734] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.734] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4eb64f2e),CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.734] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.735] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:18.735] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=92, cap=92)
[first] [DEBUG] [06/14/2012 08:19:18.735] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.735] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:18.735] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=92, cap=92)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.735] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=92, cap=92)
[first] [DEBUG] [06/14/2012 08:19:18.738] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:18.738] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:18.791] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:18.791] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:18.791] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:18.791] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:18.791] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:18.792] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:18.792] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.792] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:18.794] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.794] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6bb5eba4),CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.794] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.795] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3164, cap=3164)
[second] [DEBUG] [06/14/2012 08:19:18.795] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=3164, cap=3164)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.795] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3164, cap=3164)
[second] [DEBUG] [06/14/2012 08:19:18.798] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.798] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:18.806] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:18.809] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Cluster convergence reached: [Member(address = akka://ConvergenceSpec@fourth, status = Joining)]
[fourth] [INFO] [06/14/2012 08:19:18.809] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Leader is moving node [akka://ConvergenceSpec@fourth] from JOINING to UP
[first] [DEBUG] [06/14/2012 08:19:18.890] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.890] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:18.891] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.891] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.892] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:18.892] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4b25ee49),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.892] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.892] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.892] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:18.893] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.893] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1e37504d),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.893] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:18.893] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:18.893] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:18.894] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:18.894] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:18.895] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.895] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:18.895] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:18.895] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:18.927] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:18.927] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:18.927] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:18.927] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:18.928] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:18.928] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:18.932] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x539c5048, localhost/127.0.0.1:61435 => /127.0.0.1:61427] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:18.929] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:18.931] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x219b6225, /127.0.0.1:61435 => localhost/127.0.0.1:61427] WRITE: CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.931] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/fourth-snd4] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@38f8ff42,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@410c6406),CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.931] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/fourth-snd4] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:18.932] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x539c5048, localhost/127.0.0.1:61435 => /127.0.0.1:61427] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:18.932] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x539c5048, localhost/127.0.0.1:61435 => /127.0.0.1:61427] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=92, cap=92)
[fourth] [DEBUG] [06/14/2012 08:19:18.932] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@42bb0406,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4054824),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:18.932] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[fourth] [DEBUG] [06/14/2012 08:19:18.933] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@42bb0406,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4054824),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:18.934] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:18.934] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@42bb0406,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4054824),BigEndianHeapChannelBuffer(ridx=0, widx=92, cap=92)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:18.934] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=92, cap=92)
[fourth] [DEBUG] [06/14/2012 08:19:18.966] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [INFO] [06/14/2012 08:19:18.973] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Can't establish a causal relationship between "remote" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))] and "local" gossip [Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5619a)], unreachable = []), members = [Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5619a))] - merging them into [Gossip(overview = GossipOverview(seen = [], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))]
[fourth] [DEBUG] [06/14/2012 08:19:18.973] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:18.989] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:18.989] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:18.989] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:18.990] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:18.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:18.990] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:18.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:18.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:18.992] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[first] [DEBUG] [06/14/2012 08:19:18.992] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@34189cab),CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:18.992] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[second] [DEBUG] [06/14/2012 08:19:18.992] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[second] [DEBUG] [06/14/2012 08:19:18.992] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=92, cap=92)
[second] [DEBUG] [06/14/2012 08:19:18.993] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.993] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[second] [DEBUG] [06/14/2012 08:19:18.993] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=92, cap=92)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:18.993] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=92, cap=92)
[second] [DEBUG] [06/14/2012 08:19:18.996] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:18.996] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.004] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:19.005] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.005] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:19.005] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.006] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Adding cluster node [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.007] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.008] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:19.008] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.GossipEnvelope]
[fourth] [DEBUG] [06/14/2012 08:19:19.015] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3019, cap=3019, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.015] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@428d5aad),CompositeChannelBuffer(ridx=0, widx=3019, cap=3019, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.015] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3019, cap=3019, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.015] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:19.015] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[first] [DEBUG] [06/14/2012 08:19:19.016] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1995, cap=1995)
[first] [DEBUG] [06/14/2012 08:19:19.016] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.016] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[first] [DEBUG] [06/14/2012 08:19:19.016] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=1995, cap=1995)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.016] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1995, cap=1995)
[first] [DEBUG] [06/14/2012 08:19:19.022] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:19.023] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:19.028] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:19.028] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.028] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:19.026] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.016] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:19.026] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:19.027] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:19.027] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:19.027] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@10e5f0d2),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:19.027] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@second) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:19.028] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:19.028] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x219b6225, /127.0.0.1:61435 => localhost/127.0.0.1:61427] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:19.028] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/fourth-snd4] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@38f8ff42,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4398ee2f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:19.028] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/fourth-snd4] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.029] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:19.030] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:19.016] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:19.021] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:19.022] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:19.022] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:19.024] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4e3c35fd] WRITE: CompositeChannelBuffer(ridx=0, widx=3019, cap=3019, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.024] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3019, cap=3019, components=2) in direction Send when socket [id: 0x4e3c35fd] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:19.025] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:19.026] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:19.029] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x539c5048, localhost/127.0.0.1:61435 => /127.0.0.1:61427] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:19.029] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@42bb0406,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4054824),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.029] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:19.030] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@second) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:19.031] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:19.031] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[second] [DEBUG] [06/14/2012 08:19:19.124] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Initiating new round of gossip
[second] [DEBUG] [06/14/2012 08:19:19.125] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:19.125] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:19.125] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:19.125] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:19.125] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:19.126] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:19.128] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[second] [DEBUG] [06/14/2012 08:19:19.129] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Selecting random node to gossip to [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:19.129] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@675ee9e3),CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:19.130] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[second] [DEBUG] [06/14/2012 08:19:19.130] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.130] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3164, cap=3164)
[first] [DEBUG] [06/14/2012 08:19:19.130] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@1b8d4c9a),BigEndianHeapChannelBuffer(ridx=0, widx=3164, cap=3164)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.130] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3164, cap=3164)
[second] [DEBUG] [06/14/2012 08:19:19.131] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[second] [DEBUG] [06/14/2012 08:19:19.132] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x219b6225, /127.0.0.1:61435 => localhost/127.0.0.1:61427] WRITE: CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[second] [DEBUG] [06/14/2012 08:19:19.133] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/fourth-snd4] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@38f8ff42,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3f94a1f),CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:19.133] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/fourth-snd4] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3164, cap=3164, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.133] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x539c5048, localhost/127.0.0.1:61435 => /127.0.0.1:61427] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:19.133] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x539c5048, localhost/127.0.0.1:61435 => /127.0.0.1:61427] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1116, cap=1116)
[fourth] [DEBUG] [06/14/2012 08:19:19.134] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@42bb0406,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4054824),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.134] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:19.134] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:19.134] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@42bb0406,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@4054824),BigEndianHeapChannelBuffer(ridx=0, widx=1116, cap=1116)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.134] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:19.134] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-rcv2] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1116, cap=1116)
[fourth] [DEBUG] [06/14/2012 08:19:19.139] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@second,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@third -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a55b5a, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@second -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Joining)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:19.140] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:19.187] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:19.187] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:19.187] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:19.187] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:19.187] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:19.187] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:19.187] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:19.188] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:19.190] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3115, cap=3115, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.190] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7897aaa6),CompositeChannelBuffer(ridx=0, widx=3115, cap=3115, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.191] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3115, cap=3115, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.191] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[fourth] [DEBUG] [06/14/2012 08:19:19.191] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:19.191] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=43, cap=43)
[fourth] [DEBUG] [06/14/2012 08:19:19.192] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.192] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[fourth] [DEBUG] [06/14/2012 08:19:19.192] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.192] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:19.192] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=43, cap=43)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.192] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=43, cap=43)
[fourth] [DEBUG] [06/14/2012 08:19:19.199] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:19.199] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.201] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:19.202] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.202] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:19.202] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.202] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.202] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.203] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:19.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:19.207] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5a4fdf11),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.207] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:19.208] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:19.208] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.208] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1127, cap=1127)
[first] [DEBUG] [06/14/2012 08:19:19.208] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.208] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[first] [DEBUG] [06/14/2012 08:19:19.208] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=1127, cap=1127)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.208] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1127, cap=1127)
[fourth] [DEBUG] [06/14/2012 08:19:19.209] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:19.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x539c5048, localhost/127.0.0.1:61435 => /127.0.0.1:61427] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.211] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@42bb0406,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3c870e2f),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.211] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.212] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:19.212] [New I/O client worker #1-4] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x219b6225, /127.0.0.1:61435 => localhost/127.0.0.1:61427] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:19.212] [New I/O client worker #1-4] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x219b6225, /127.0.0.1:61435 => localhost/127.0.0.1:61427] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:19.212] [New I/O client worker #1-4] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x219b6225, /127.0.0.1:61435 => localhost/127.0.0.1:61427] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [DEBUG] [06/14/2012 08:19:19.212] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[second] [DEBUG] [06/14/2012 08:19:19.212] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/fourth-rcv4] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@38f8ff42,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@26a0c73f),BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:19.213] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/fourth-rcv4] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1024, cap=1024)
[second] [DEBUG] [06/14/2012 08:19:19.213] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/fourth-rcv4] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@38f8ff42,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@26a0c73f),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:19.213] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/fourth-rcv4] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[second] [DEBUG] [06/14/2012 08:19:19.213] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/fourth-rcv4] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@38f8ff42,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@26a0c73f),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[second] [DEBUG] [06/14/2012 08:19:19.213] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/fourth-rcv4] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[second] [DEBUG] [06/14/2012 08:19:19.221] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:19.222] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:19.286] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:19.286] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:19.287] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a690247, localhost/127.0.0.1:61430 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.287] [ConvergenceSpec-akka.actor.default-dispatcher-2] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:19.287] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4adf9095,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7a120cb3),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.288] [ConvergenceSpec-akka.actor.default-dispatcher-2] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:19.288] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61430-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.288] [New I/O server worker #2-2] [ConductorHandler(akka://ConvergenceSpec)] disconnect from /127.0.0.1:61424
[second] [DEBUG] [06/14/2012 08:19:19.288] [ConvergenceSpec-akka.actor.default-dispatcher-4] [PlayerHandler(akka://ConvergenceSpec)] disconnected from localhost/127.0.0.1:4711
[second] [DEBUG] [06/14/2012 08:19:19.288] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x72ebf3f0, /127.0.0.1:61430 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:19.288] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@2f7fc44f,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@32b95f72),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.288] [ConvergenceSpec-akka.actor.default-dispatcher-2] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[second] [DEBUG] [06/14/2012 08:19:19.289] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:19.289] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:19.290] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.289] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/user/controller/127.0.0.1:61424-server2] processing Event(ClientDisconnected,None) from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:19.289] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2a06bbe7),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.289] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.290] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[second] [DEBUG] [06/14/2012 08:19:19.290] [ConvergenceSpec-akka.actor.default-dispatcher-4] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 :> localhost/127.0.0.1:4711] unbound
[second] [DEBUG] [06/14/2012 08:19:19.290] [ConvergenceSpec-akka.actor.default-dispatcher-4] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0f786a3c, /127.0.0.1:61424 :> localhost/127.0.0.1:4711] closed
[second] [WARN] [06/14/2012 08:19:19.290] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/user/TestConductorClient] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: PoisonPill
[second] [DEBUG] [06/14/2012 08:19:19.290] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@second)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@second/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[second] [DEBUG] [06/14/2012 08:19:19.290] [ConvergenceSpec-akka.actor.default-dispatcher-2] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.291] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:19.291] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[second] [DEBUG] [06/14/2012 08:19:19.292] [ConvergenceSpec-akka.actor.default-dispatcher-2] [EventStream] shutting down: StandardOutLogger started
[second] [DEBUG] [06/14/2012 08:19:19.292] [ConvergenceSpec-akka.actor.default-dispatcher-2] [EventStream] shutting down: StandardOutLogger started
[second] [DEBUG] [06/14/2012 08:19:19.293] [ConvergenceSpec-akka.actor.default-dispatcher-2] [EventStream] all default loggers stopped
[second] [INFO] [06/14/2012 08:19:19.298] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@second] - Shutting down cluster Node and cluster daemons...
[second] - not move a new joining node to Up while there is no convergence *** FAILED ***
[second]   Some(Up) was not equal to Some(Joining) (ConvergenceSpec.scala:104)
[second]   org.scalatest.TestFailedException:
[second]   at org.scalatest.matchers.Matchers$class.newTestFailedException(Matchers.scala:150)
[second]   at akka.testkit.AkkaSpec.newTestFailedException(AkkaSpec.scala:59)
[second]   at org.scalatest.matchers.MustMatchers$MustMethodHelper$.mustMatcher(MustMatchers.scala:873)
[second]   at org.scalatest.matchers.MustMatchers$AnyRefMustWrapper.must(MustMatchers.scala:1491)
[second]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$assertNotMovedUp$1$1.apply$mcV$sp(ConvergenceSpec.scala:104)
[second]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$assertNotMovedUp$1$1.apply(ConvergenceSpec.scala:98)
[second]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$assertNotMovedUp$1$1.apply(ConvergenceSpec.scala:98)
[second]   at akka.testkit.TestKitBase$class.within(TestKit.scala:222)
[second]   at akka.testkit.TestKit.within(TestKit.scala:600)
[second]   at akka.testkit.TestKitBase$class.within(TestKit.scala:236)
[second]   at akka.testkit.TestKit.within(TestKit.scala:600)
[second]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.assertNotMovedUp$1(ConvergenceSpec.scala:98)
[second]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$apply$mcV$sp$13$$anonfun$apply$mcV$sp$14.apply$mcVI$sp(ConvergenceSpec.scala:113)
[second]   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
[second]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$apply$mcV$sp$13.apply$mcV$sp(ConvergenceSpec.scala:111)
[second]   at akka.remote.testkit.MultiNodeSpec.runOn(MultiNodeSpec.scala:177)
[second]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.apply$mcV$sp(ConvergenceSpec.scala:110)
[second]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.apply(ConvergenceSpec.scala:84)
[second]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.apply(ConvergenceSpec.scala:84)
[second]   at org.scalatest.WordSpec$$anon$2.apply(WordSpec.scala:2161)
[second]   at org.scalatest.Suite$class.withFixture(Suite.scala:1968)
[second]   at akka.testkit.AkkaSpec.withFixture(AkkaSpec.scala:59)
[second]   at org.scalatest.WordSpec$class.invokeWithFixture$1(WordSpec.scala:2158)
[second]   at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[second]   at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[second]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:168)
[second]   at org.scalatest.WordSpec$class.runTest(WordSpec.scala:2167)
[second]   at akka.testkit.AkkaSpec.runTest(AkkaSpec.scala:59)
[second]   at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[second]   at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[second]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:226)
[second]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[second]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[second]   at scala.collection.immutable.List.foreach(List.scala:76)
[second]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[second]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:231)
[second]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[second]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[second]   at scala.collection.immutable.List.foreach(List.scala:76)
[second]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[second]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:282)
[second]   at org.scalatest.WordSpec$class.runTests(WordSpec.scala:2232)
[second]   at akka.testkit.AkkaSpec.runTests(AkkaSpec.scala:59)
[second]   at org.scalatest.Suite$class.run(Suite.scala:2286)
[second]   at akka.testkit.AkkaSpec.org$scalatest$WordSpec$$super$run(AkkaSpec.scala:59)
[second]   at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[second]   at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[second]   at org.scalatest.SuperEngine.runImpl(Engine.scala:318)
[second]   at org.scalatest.WordSpec$class.run(WordSpec.scala:2279)
[second]   at akka.testkit.AkkaSpec.org$scalatest$BeforeAndAfterAll$$super$run(AkkaSpec.scala:59)
[second]   at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:213)
[second]   at akka.testkit.AkkaSpec.run(AkkaSpec.scala:59)
[second]   at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:59)
[second]   at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1517)
[second]   at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1514)
[second]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[second]   at scala.collection.immutable.List.foreach(List.scala:76)
[second]   at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1514)
[second]   at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:584)
[second]   at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:583)
[second]   at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1558)
[second]   at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:582)
[second]   at org.scalatest.tools.Runner$.main(Runner.scala:466)
[second]   at org.scalatest.tools.Runner.main(Runner.scala)
[second]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[second]   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[second]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[second]   at java.lang.reflect.Method.invoke(Method.java:597)
[second]   at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
[second]   at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
[second]   at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
[second]   at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
[second]   at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
[second]   at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
[second]   at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
[second]   at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
[second]   at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
[second]   at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
[second]   at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[second] [DEBUG] [06/14/2012 08:19:19.303] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@first]
[second] [INFO] [06/14/2012 08:19:19.303] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@first
[second] [DEBUG] [06/14/2012 08:19:19.303] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x72ebf3f0, /127.0.0.1:61430 :> localhost/127.0.0.1:61416]
[second] [DEBUG] [06/14/2012 08:19:19.304] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@first] has been shut down
[second] [DEBUG] [06/14/2012 08:19:19.304] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@fourth]
[second] [INFO] [06/14/2012 08:19:19.304] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@fourth
[fourth] [DEBUG] [06/14/2012 08:19:19.302] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.303] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:19.303] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.serialization.Serialization(akka://ConvergenceSpec)] Using serializer[akka.serialization.JavaSerializer] for message [akka.cluster.Heartbeat]
[fourth] [DEBUG] [06/14/2012 08:19:19.303] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.304] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.304] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:19.304] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x539c5048, localhost/127.0.0.1:61435 => /127.0.0.1:61427] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.305] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x4a690247, localhost/127.0.0.1:61430 :> /127.0.0.1:61416]
[first] [DEBUG] [06/14/2012 08:19:19.305] [ConvergenceSpec-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [PassiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:19.305] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:19.305] [ConvergenceSpec-4] [RemoteClient(akka://ConvergenceSpec)] [PassiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:19.306] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteServerClientClosed@akka://ConvergenceSpec@first: Client[akka://ConvergenceSpec@second]
[second] [DEBUG] [06/14/2012 08:19:19.306] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x219b6225, /127.0.0.1:61435 :> localhost/127.0.0.1:61427]
[second] [DEBUG] [06/14/2012 08:19:19.306] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@fourth] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:19.309] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-snd2] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@42bb0406,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@30e33af5),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.309] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61435-snd2] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.309] [New I/O server worker #1-1] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x539c5048, localhost/127.0.0.1:61435 :> /127.0.0.1:61427]
[fourth] [DEBUG] [06/14/2012 08:19:19.309] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@32486cdd),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.310] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.310] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:19.311] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.311] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:19.312] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:19.313] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [WARN] [06/14/2012 08:19:19.314] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec@second/system/cluster/gossip] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: Heartbeat(akka://ConvergenceSpec@fourth)
[fourth] [ERROR] [06/14/2012 08:19:19.314] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteServerError@akka://ConvergenceSpec@fourth] Error[java.nio.channels.ClosedChannelException]
[fourth] [DEBUG] [06/14/2012 08:19:19.316] [ConvergenceSpec-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [PassiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:19.316] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:19.316] [ConvergenceSpec-6] [RemoteClient(akka://ConvergenceSpec)] [PassiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:19.318] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteServerClientClosed@akka://ConvergenceSpec@fourth: Client[akka://ConvergenceSpec@second]
[second] [INFO] [06/14/2012 08:19:19.333] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteServerShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:19.385] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:19.386] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:19.386] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:19.386] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:19.390] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.390] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@51af7c57),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.390] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.390] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:19.391] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=1127, cap=1127)
[fourth] [DEBUG] [06/14/2012 08:19:19.391] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.391] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=2048, cap=2048)
[fourth] [DEBUG] [06/14/2012 08:19:19.391] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=1127, cap=1127)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.391] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=1127, cap=1127)
[fourth] [DEBUG] [06/14/2012 08:19:19.394] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:19.395] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.400] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:19.400] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.400] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:19.401] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:19.402] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:19.403] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:19.404] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:19.408] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3ab28980] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.408] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3ab28980] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:19.408] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:19.409] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:19.409] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:19.418] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:19.418] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:19.418] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:19.418] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:19.597] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:19.598] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.598] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:19.598] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.598] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.598] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.599] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:19.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:19.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.601] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@455b4492),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.602] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:19.601] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.602] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [DEBUG] [06/14/2012 08:19:19.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:19.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:19.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:19.603] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:19.604] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:19.604] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:19.605] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:19.605] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.605] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6af2f0d0] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.606] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6af2f0d0] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:19.606] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:19.606] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:19.606] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:19.616] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:19.616] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:19.616] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:19.616] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:19.616] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:19.616] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:19.617] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:19.619] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:19.627] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:19.627] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:19.628] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:19.630] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x617e53c9] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.631] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x617e53c9] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:19.631] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:19.631] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:19.634] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:19.697] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.697] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:19.698] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.698] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.698] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:19.698] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1bb25a82),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.698] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:19.698] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.699] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.699] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:19.699] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:19.699] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:19.699] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:19.700] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5a541b02] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.700] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:19.700] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x5a541b02] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:19.700] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.700] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:19.700] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:19.700] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:19.715] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:19.716] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:19.716] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:19.717] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:19.717] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:19.717] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43fd360b] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.717] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x43fd360b] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:19.718] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:19.718] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:19.718] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:19.718] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:19.718] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:19.718] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.718] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@64c47498),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.718] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.719] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:19.719] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.719] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:19.720] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:19.720] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.796] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:19.796] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.796] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:19.797] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.797] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.797] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.798] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:19.801] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.801] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@31bd669d),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.801] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.802] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:19.802] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.802] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:19.807] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:19.807] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:19.814] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:19.814] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:19.814] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:19.814] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:19.814] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:19.815] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:19.815] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:19.815] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:19.817] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:19.817] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@32bed1fd),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:19.817] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:19.817] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:19.818] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:19.818] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.818] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:19.818] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:19.818] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:19.822] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:19.822] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.994] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:19.994] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:19.995] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:19.995] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.995] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:19.996] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:19.996] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:19.996] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:19.997] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:19.998] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.001] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4a4de3fe] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.001] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:20.001] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x4a4de3fe] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:20.001] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:20.001] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:20.001] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.002] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:20.002] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:20.002] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:20.002] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:20.003] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.005] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x144f3ba2] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.006] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x144f3ba2] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:20.006] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:20.006] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:20.006] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.012] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:20.012] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:20.013] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:20.013] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.013] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.013] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:20.013] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:20.013] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:20.014] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.015] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@33cfa965),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:20.015] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.015] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:20.015] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:20.015] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:20.018] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:20.019] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:20.093] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.093] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.094] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.094] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.095] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:20.095] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6d87275d),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:20.095] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.096] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:20.096] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.096] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:20.096] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:20.096] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:20.097] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:20.097] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x442982d8] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.097] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x442982d8] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:20.098] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:20.098] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:20.098] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:20.098] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.098] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.111] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:20.111] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:20.112] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.114] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:20.114] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.114] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x067a4076] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.116] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:20.116] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x067a4076] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:20.116] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:20.116] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:20.116] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.116] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:20.117] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.117] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6fa37fac),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:20.117] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.117] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:20.118] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:20.118] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:20.119] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:20.119] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:20.192] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:20.193] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:20.193] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:20.194] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:20.194] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:20.194] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.195] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.198] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.198] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:20.200] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.200] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@28f549de),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:20.200] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.200] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:20.200] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:20.201] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:20.201] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.201] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:20.201] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:20.203] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.205] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7fc4a2d3] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.205] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x7fc4a2d3] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:20.205] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:20.205] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:20.205] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.205] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:20.206] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.210] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:20.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:20.211] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:20.211] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:20.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.211] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:20.212] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:20.214] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.214] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@12e29f36),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:20.214] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.214] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:20.215] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:20.215] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:20.217] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:20.218] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:20.390] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:20.391] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:20.391] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:20.391] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:20.392] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:20.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:20.393] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:20.393] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:20.395] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.397] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2f8fc9ba] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.398] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:20.399] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2f8fc9ba] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:20.399] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:20.399] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:20.409] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:20.409] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:20.409] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.409] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:20.410] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:20.410] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:20.412] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:20.412] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.413] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:20.415] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x62f04dae] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.415] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x62f04dae] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:20.415] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:20.415] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:20.415] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.415] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:20.415] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:20.415] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:20.417] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:20.417] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.418] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:20.419] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x174da86b] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.419] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x174da86b] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:20.419] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:20.419] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:20.419] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:20.489] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.489] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.490] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.490] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.491] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:20.492] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4d0ac2a3),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:20.492] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.492] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:20.493] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:20.493] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.493] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:20.493] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:20.493] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:20.494] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7d7082d8] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.494] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:20.494] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x7d7082d8] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:20.495] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:20.495] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:20.495] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:20.495] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.507] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:20.507] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:20.508] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.508] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:20.509] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.509] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3aa42c31] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.509] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x3aa42c31] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:20.509] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:20.509] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:20.509] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:20.510] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.510] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:20.510] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.510] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1a15cd9a),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:20.510] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.511] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:20.511] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:20.511] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:20.512] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:20.512] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:20.588] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:20.589] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:20.589] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:20.590] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:20.590] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:20.591] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.591] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.594] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.594] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:20.594] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.595] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@56cdd54e),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:20.595] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.595] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:20.595] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:20.595] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [DEBUG] [06/14/2012 08:19:20.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:20.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [ERROR] [06/14/2012 08:19:20.596] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:20.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [ERROR] [06/14/2012 08:19:20.596] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:20.597] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.599] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x636f2067] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.599] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x636f2067] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:20.599] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:20.599] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:20.599] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.600] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:20.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.605] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:20.606] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:20.606] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:20.606] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.606] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.606] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:20.606] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:20.606] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:20.607] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:20.607] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.607] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:20.609] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0f6a8823] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.609] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:20.609] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x0f6a8823] already closed, dropping
[first] [INFO] [06/14/2012 08:19:20.609] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:20.609] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:20.786] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:20.786] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:20.787] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:20.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:20.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:20.788] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.788] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:20.788] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:20.789] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:20.790] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.792] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2b347d33] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.793] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2b347d33] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:20.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:20.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:20.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.804] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:20.804] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:20.804] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.804] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:20.804] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:20.805] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:20.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:20.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:20.806] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.807] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2bf87baf),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:20.807] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.807] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:20.807] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:20.808] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:20.810] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:20.811] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:20.885] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.885] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.885] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.886] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.886] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@62cc9d70),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:20.886] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.886] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:20.886] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:20.887] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:20.887] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:20.887] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:20.887] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [ERROR] [06/14/2012 08:19:20.886] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:20.888] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:20.888] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:20.888] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x677456bc] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.888] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x677456bc] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:20.888] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:20.889] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:20.889] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.903] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:20.903] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:20.904] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.904] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:20.904] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:20.905] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x468034b6] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.905] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x468034b6] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:20.905] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:20.905] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:20.905] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:20.905] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:20.905] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:20.905] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:20.906] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6fb8fcdd),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:20.906] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.906] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:20.906] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:20.906] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:20.907] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:20.907] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:20.983] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:20.983] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:20.984] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:20.984] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:20.984] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:20.984] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:20.984] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:20.985] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:20.985] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:20.986] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:20.987] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x447bbfb8] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:20.987] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x447bbfb8] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:20.987] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:20.987] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:20.988] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:21.001] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:21.002] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:21.002] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:21.002] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.002] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.002] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:21.002] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:21.003] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:21.004] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.004] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:21.004] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:21.004] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4e850e8c),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:21.004] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.005] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:21.005] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:21.005] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:21.005] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:21.005] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:21.005] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [ERROR] [06/14/2012 08:19:21.005] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:21.005] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [ERROR] [06/14/2012 08:19:21.006] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:21.006] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:21.007] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x017e319a] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.008] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x017e319a] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:21.008] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:21.008] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:21.008] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:21.009] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:21.009] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:21.181] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:21.181] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:21.182] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:21.182] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:21.182] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:21.182] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:21.182] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:21.184] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.184] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:21.184] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:21.184] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@421906df),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:21.184] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.185] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:21.185] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:21.185] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:21.185] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:21.185] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:21.185] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:21.186] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:21.187] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x79123c5f] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.187] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x79123c5f] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:21.187] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:21.188] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:21.188] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:21.188] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:21.188] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.199] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:21.200] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:21.200] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:21.200] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.200] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.200] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:21.200] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:21.200] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:21.202] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.202] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:21.202] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:21.202] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3403f5a0),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:21.202] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.202] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:21.202] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:21.203] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:21.203] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:21.203] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:21.203] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:21.204] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:21.205] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6ff2f60d] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.205] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6ff2f60d] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:21.205] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:21.205] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:21.205] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:21.206] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:21.207] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:21.281] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:21.281] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:21.282] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.282] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:21.282] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:21.283] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7a73bc22),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:21.283] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.283] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:21.283] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:21.283] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:21.284] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:21.284] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:21.284] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:21.284] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x44449afc] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.285] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x44449afc] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:21.285] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:21.285] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:21.285] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:21.285] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:21.285] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:21.299] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:21.300] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:21.301] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:21.301] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:21.302] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:21.302] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2025b64d] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.303] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:21.304] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x2025b64d] already closed, dropping
[first] [INFO] [06/14/2012 08:19:21.304] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:21.304] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:21.304] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:21.304] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:21.305] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.305] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@55444319),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:21.305] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.306] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:21.306] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:21.306] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:21.308] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:21.308] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:21.379] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:21.379] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:21.379] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:21.380] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:21.380] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:21.380] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:21.380] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:21.382] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.382] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:21.383] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:21.383] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@899ff91),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:21.383] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.383] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:21.383] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:21.383] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:21.383] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:21.383] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:21.384] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:21.384] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:21.385] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x36598d00] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.386] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x36598d00] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:21.386] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:21.386] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:21.386] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:21.387] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:21.387] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.398] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:21.398] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:21.398] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:21.398] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.398] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:21.398] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.398] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:21.398] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:21.399] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:21.399] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:21.400] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:21.401] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x41d753b7] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.401] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x41d753b7] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:21.401] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:21.401] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:21.401] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:21.578] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:21.579] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:21.579] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:21.579] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:21.579] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:21.579] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:21.579] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:21.581] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:21.582] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:21.583] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:21.584] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x23bb9829] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.584] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x23bb9829] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:21.584] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:21.585] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:21.585] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:21.595] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:21.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:21.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:21.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.596] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.596] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:21.596] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:21.596] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:21.597] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:21.597] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:21.598] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:21.599] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x377e4cec] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.600] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x377e4cec] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:21.600] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:21.600] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:21.600] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:21.676] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:21.677] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:21.677] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.677] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:21.677] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@58779959),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:21.677] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:21.677] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.678] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:21.678] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:21.678] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:21.678] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:21.679] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:21.679] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:21.679] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:21.679] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:21.679] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x13f4cf6f] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.679] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x13f4cf6f] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:21.679] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:21.680] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:21.680] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:21.695] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:21.696] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:21.697] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:21.698] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:21.698] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:21.698] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x27e3bfb6] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.698] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x27e3bfb6] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:21.699] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:21.699] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:21.699] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:21.699] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:21.699] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:21.699] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.700] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@529801f4),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:21.700] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.700] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:21.701] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:21.701] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:21.702] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:21.703] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:21.776] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:21.776] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:21.776] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:21.776] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:21.777] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:21.777] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:21.778] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:21.779] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6913108b] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.780] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6913108b] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:21.780] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:21.780] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:21.781] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:21.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:21.794] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:21.794] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:21.794] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.794] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.794] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:21.794] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:21.794] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:21.795] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.796] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4638fb59),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:21.796] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:21.796] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:21.797] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:21.797] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:21.799] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:21.800] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:21.809] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:21.809] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:21.810] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.992] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:21.992] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:21.993] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:21.993] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.993] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:21.993] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:21.994] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:21.994] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:21.995] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:21.996] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:21.997] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:21.999] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x10db5b3f] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:21.999] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x10db5b3f] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:21.999] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:21.999] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:22.000] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:22.007] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:22.007] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.007] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:22.007] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.007] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.008] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.008] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:22.010] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.010] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:22.010] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.011] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2c164804),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.011] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:22.011] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.011] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:22.011] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:22.012] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:22.012] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:22.012] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:22.013] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:22.014] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6ad21538] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.014] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6ad21538] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:22.014] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:22.014] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:22.014] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:22.015] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:22.016] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:22.091] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:22.091] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:22.093] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:22.093] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:22.093] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:22.094] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4c84f665] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.094] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x4c84f665] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:22.094] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:22.094] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:22.094] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:22.095] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:22.095] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:22.095] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.095] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@22e1469c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:22.096] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.096] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:22.096] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.097] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:22.098] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:22.098] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.106] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.107] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:22.107] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.107] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.107] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6d3d7254),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.107] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:22.108] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.108] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:22.108] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:22.108] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:22.108] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:22.108] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:22.109] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:22.109] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5d1d20d3] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.109] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x5d1d20d3] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:22.109] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:22.109] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:22.109] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:22.109] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.109] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:22.190] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:22.191] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:22.191] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:22.191] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:22.191] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:22.192] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:22.192] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:22.193] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:22.195] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.195] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@dc737be),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:22.195] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.196] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:22.196] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.196] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:22.201] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:22.202] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.205] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:22.205] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.205] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:22.205] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.206] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.206] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.206] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:22.206] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:22.207] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:22.208] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:22.209] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x697a906e] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.209] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x697a906e] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:22.210] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:22.210] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:22.210] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:22.388] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:22.388] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:22.389] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:22.389] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:22.389] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:22.389] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:22.389] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:22.390] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:22.391] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.391] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:22.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:22.392] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@14f3cf72),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:22.392] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:22.392] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:22.392] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.392] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:22.393] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:22.394] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:22.394] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:22.395] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x12b8501d] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.396] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x12b8501d] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:22.396] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:22.396] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [INFO] [06/14/2012 08:19:22.396] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:22.396] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:22.396] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.403] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:22.403] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.403] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:22.403] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.404] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.404] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.404] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:22.405] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.405] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:22.405] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.406] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1e37504d),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.406] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.406] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:22.406] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:22.406] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:22.406] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:22.406] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:22.407] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:22.407] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:22.408] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4178feba] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.408] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x4178feba] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:22.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:22.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:22.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:22.409] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:22.410] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:22.487] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:22.488] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:22.489] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:22.489] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:22.489] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:22.490] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1330b13c] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.490] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x1330b13c] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:22.491] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:22.491] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:22.491] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:22.491] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:22.492] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:22.492] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.492] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3e0339),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:22.492] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.493] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:22.493] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.493] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:22.494] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:22.494] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.502] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.502] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:22.502] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.502] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.502] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4c767fb3),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.503] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.503] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:22.503] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:22.503] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:22.503] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:22.503] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:22.504] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:22.504] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:22.504] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x77b9e7fc] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.504] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x77b9e7fc] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:22.504] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:22.504] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:22.504] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [INFO] [06/14/2012 08:19:22.504] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:22.505] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:22.586] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:22.587] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:22.587] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:22.587] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:22.587] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:22.587] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:22.587] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:22.587] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:22.588] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:22.588] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:22.589] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:22.590] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0ca8e2b8] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.591] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x0ca8e2b8] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:22.591] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:22.591] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:22.591] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:22.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:22.602] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.602] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:22.602] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.603] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:22.605] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.605] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@675926d1),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.605] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.606] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:22.606] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:22.606] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:22.610] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:22.610] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:22.799] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.799] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:22.799] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.799] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.799] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.800] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:22.801] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.801] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:22.801] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.801] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@53b258fa),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.801] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.801] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:22.802] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:22.802] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:22.802] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:22.803] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:22.803] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:22.804] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:22.805] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:22.805] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x61efb003] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.806] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.806] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x61efb003] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:22.806] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:22.806] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:22.806] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:22.817] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:22.817] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:22.817] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:22.817] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:22.817] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:22.817] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:22.817] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:22.818] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:22.819] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.819] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3801ff83),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:22.819] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.819] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:22.826] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.830] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:22.834] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:22.834] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.898] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.899] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:22.899] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.899] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.899] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:22.899] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7669521),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.899] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.899] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:22.900] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:22.900] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:22.900] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:22.900] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:22.900] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:22.901] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:22.901] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.900] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5fc8ad52] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.902] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x5fc8ad52] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:22.902] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:22.902] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:22.902] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:22.916] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:22.916] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:22.917] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:22.918] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:22.918] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:22.918] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0b2e0e2f] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.918] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:22.918] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x0b2e0e2f] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:22.918] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:22.918] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:22.919] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:22.919] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:22.919] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:22.919] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@ea224b5),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:22.919] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:22.919] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:22.919] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:22.920] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:22.920] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:22.920] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.997] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:22.997] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:22.997] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:22.997] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.998] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:22.998] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:22.998] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:22.998] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:23.000] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:23.000] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:23.001] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x29d602b3] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.003] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:23.003] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:23.003] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x29d602b3] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:23.003] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:23.003] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.003] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:23.004] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:23.004] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:23.004] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:23.005] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:23.006] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6fe88c7f] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.007] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6fe88c7f] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:23.007] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:23.007] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:23.007] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:23.015] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:23.015] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:23.015] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:23.015] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:23.015] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:23.016] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:23.016] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:23.016] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:23.017] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.017] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:23.017] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:23.018] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7d01acff),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:23.018] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.018] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:23.018] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:23.018] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:23.018] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:23.018] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:23.019] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:23.019] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:23.020] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7904d976] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.021] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x7904d976] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:23.021] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:23.021] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [INFO] [06/14/2012 08:19:23.022] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:23.022] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:23.022] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:23.195] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:23.195] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:23.195] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:23.195] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:23.196] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:23.196] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.196] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:23.198] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.198] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:23.198] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.198] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@f4c0275),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:23.199] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.199] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:23.199] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:23.199] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [DEBUG] [06/14/2012 08:19:23.199] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:23.200] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:23.200] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:23.200] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [ERROR] [06/14/2012 08:19:23.199] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:23.201] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:23.202] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:23.203] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:23.203] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4735572b] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.203] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x4735572b] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:23.203] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:23.203] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [INFO] [06/14/2012 08:19:23.203] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:23.203] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:23.213] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:23.213] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:23.213] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:23.213] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:23.213] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:23.213] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:23.213] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:23.213] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:23.215] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:23.215] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:23.216] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:23.217] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x308c666a] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.217] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:23.217] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x308c666a] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:23.217] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:23.217] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:23.217] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:23.217] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:23.218] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:23.218] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:23.218] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:23.219] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:23.220] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2214c7cd] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.220] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2214c7cd] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:23.220] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:23.220] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:23.220] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:23.294] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.294] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:23.295] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.295] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.295] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:23.296] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5782b9b5),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:23.296] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.296] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:23.297] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:23.297] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:23.297] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:23.297] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:23.298] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:23.298] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x476e46f5] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.299] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:23.299] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:23.298] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x476e46f5] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:23.300] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:23.300] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:23.300] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:23.312] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:23.313] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:23.313] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:23.313] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:23.314] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:23.314] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x58dca3ed] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.314] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x58dca3ed] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:23.314] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:23.314] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:23.314] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:23.314] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:23.314] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:23.315] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.315] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6a0239f6),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:23.315] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.315] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:23.315] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:23.315] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:23.316] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:23.316] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:23.392] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:23.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:23.393] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:23.393] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:23.393] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:23.393] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.393] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:23.394] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.395] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@45a84b38),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:23.395] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.395] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:23.395] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:23.395] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:23.398] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:23.398] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:23.411] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:23.411] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:23.411] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:23.411] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:23.411] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:23.412] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:23.412] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:23.412] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:23.412] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:23.413] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:23.413] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:23.414] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x59c83354] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.414] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x59c83354] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:23.414] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:23.415] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:23.415] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:23.590] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:23.591] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:23.591] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:23.591] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:23.591] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:23.591] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.592] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:23.592] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:23.594] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:23.595] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:23.596] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x17db8f8e] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.596] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x17db8f8e] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:23.596] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:23.597] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:23.597] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:23.609] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:23.609] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:23.609] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:23.609] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:23.609] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:23.609] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:23.610] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:23.610] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:23.611] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.612] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@413803ba),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:23.612] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.612] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:23.612] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:23.612] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:23.612] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:23.612] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:23.612] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:23.616] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:23.616] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:23.690] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.690] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:23.690] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.690] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.691] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@293a985),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:23.691] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:23.691] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.691] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:23.691] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:23.691] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:23.692] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:23.692] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [ERROR] [06/14/2012 08:19:23.691] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:23.693] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:23.693] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:23.693] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2c5e5c15] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.693] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x2c5e5c15] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:23.693] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:23.693] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:23.693] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:23.709] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:23.709] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:23.711] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:23.712] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:23.712] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:23.713] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x447cc9c9] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.713] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:23.713] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:23.713] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x447cc9c9] already closed, dropping
[first] [INFO] [06/14/2012 08:19:23.713] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:23.713] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:23.713] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:23.714] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.714] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@c33aa9b),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:23.714] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.714] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:23.714] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:23.714] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:23.715] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:23.715] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:23.788] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:23.788] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:23.789] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:23.789] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:23.789] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:23.789] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.789] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:23.790] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:23.790] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:23.790] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:23.792] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3bbe9a50] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:23.793] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3bbe9a50] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:23.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:23.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:23.807] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:23.807] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:23.807] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:23.807] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:23.807] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:23.808] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:23.808] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:23.808] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:23.808] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:23.810] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:23.810] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:23.812] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2c7103af] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:23.812] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2c7103af] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:23.812] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:23.812] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:23.812] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:23.987] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:23.987] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:23.987] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:23.987] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:23.988] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:23.988] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.988] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:23.990] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.990] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:23.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1d71bc3f),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:23.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.990] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:23.990] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:23.990] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:23.990] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:23.991] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:23.991] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:23.991] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:23.992] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:23.993] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x15fddb33] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:23.993] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x15fddb33] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:23.993] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:23.993] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:23.993] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:23.993] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:23.994] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.004] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:24.005] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:24.005] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:24.005] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.005] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:24.005] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.005] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:24.005] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:24.006] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:24.006] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.014] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.015] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x15f6ae4d] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.015] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x15f6ae4d] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:24.015] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:24.015] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:24.015] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:24.086] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.086] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:24.086] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.086] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.086] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:24.086] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1f0812ac),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.086] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.087] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:24.087] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:24.087] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.087] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:24.087] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:24.087] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:24.087] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4b01ea1e] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.088] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x4b01ea1e] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:24.088] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:24.088] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [INFO] [06/14/2012 08:19:24.088] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:24.088] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.088] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:24.104] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:24.104] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:24.105] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.105] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:24.105] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.105] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x37af78ce] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.105] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x37af78ce] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:24.105] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:24.106] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:24.106] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:24.106] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:24.106] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.106] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.106] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@556ca3d5),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:24.106] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.106] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:24.107] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.107] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:24.107] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:24.107] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:24.185] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:24.185] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:24.185] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:24.185] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.185] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.185] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.186] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:24.187] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.187] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:24.188] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@257ccb2f),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.188] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.188] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.188] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:24.188] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:24.188] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:24.188] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:24.189] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:24.190] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:24.190] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.192] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:24.192] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.192] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x23ff8de3] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.193] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:24.193] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x23ff8de3] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:24.193] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:24.194] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:24.194] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:24.194] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.194] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:24.194] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:24.194] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:24.195] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:24.196] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5fdbe72f] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.197] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x5fdbe72f] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:24.197] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:24.197] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:24.197] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:24.203] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:24.204] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:24.204] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:24.204] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.204] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.204] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:24.204] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:24.204] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:24.205] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:24.206] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.207] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5a47eaec] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.208] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:24.208] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x5a47eaec] already closed, dropping
[first] [INFO] [06/14/2012 08:19:24.208] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:24.208] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:24.382] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:24.383] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:24.383] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.383] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:24.383] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.384] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.384] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:24.384] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:24.385] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:24.386] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:24.387] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7da2467a] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.387] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x7da2467a] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:24.387] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:24.387] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:24.387] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:24.400] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:24.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:24.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:24.401] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.401] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:24.401] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:24.401] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.402] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.403] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5b44dc7b),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:24.403] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.403] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:24.403] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.403] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:24.406] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:24.407] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:24.482] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.482] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:24.482] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.482] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.483] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:24.483] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@464e32c8),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.483] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.483] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:24.483] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.483] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.483] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.483] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:24.483] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:24.484] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x37e80c87] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.484] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:24.484] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x37e80c87] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:24.484] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:24.484] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:24.484] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:24.484] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.500] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:24.500] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:24.502] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.502] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:24.503] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.503] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x05e73e0d] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.504] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x05e73e0d] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:24.504] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:24.504] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:24.504] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:24.504] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:24.504] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.505] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.505] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6040f3c1),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:24.505] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.506] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:24.506] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.506] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:24.507] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:24.508] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:24.581] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:24.581] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:24.581] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:24.581] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.581] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.581] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.582] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:24.583] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.583] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:24.583] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.583] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@12462631),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.583] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.583] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:24.584] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:24.584] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:24.584] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:24.584] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:24.584] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:24.585] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:24.586] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3b756456] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.586] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3b756456] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:24.586] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:24.586] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:24.586] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:24.587] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:24.587] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.599] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:24.599] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:24.599] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:24.599] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.599] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.599] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:24.599] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:24.600] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:24.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:24.601] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@78a40f0e),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:24.601] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:24.601] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:24.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:24.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:24.603] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.604] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:24.605] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:24.606] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:24.605] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5ad89e02] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.606] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x5ad89e02] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:24.606] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:24.606] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:24.606] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:24.778] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:24.779] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:24.779] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:24.779] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.779] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.779] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.779] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:24.779] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:24.780] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:24.780] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:24.781] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6c4bd767] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.781] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6c4bd767] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:24.782] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:24.782] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:24.782] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:24.797] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:24.797] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:24.798] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:24.798] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.798] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:24.799] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:24.799] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:24.800] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:24.800] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.801] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.803] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2771f735] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.803] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2771f735] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:24.803] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:24.804] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:24.804] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:24.878] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.878] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:24.879] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.879] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.879] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:24.879] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@65370ce9),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.879] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.879] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:24.879] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.879] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.879] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.880] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:24.880] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:24.880] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0130c8e9] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.880] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x0130c8e9] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:24.880] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:24.880] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:24.880] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:24.880] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:24.880] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.896] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:24.896] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:24.897] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.897] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:24.897] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:24.897] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4f50f0e2] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.898] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x4f50f0e2] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:24.898] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:24.898] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:24.898] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:24.898] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:24.898] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.898] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.898] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@26c94114),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:24.898] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.899] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:24.899] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.899] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:24.900] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:24.900] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:24.977] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:24.977] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:24.978] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:24.978] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.978] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:24.979] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:24.979] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:24.981] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.982] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@40a16605),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.982] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.982] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:24.983] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:24.983] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:24.988] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:24.988] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.995] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:24.995] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:24.995] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:24.995] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:24.996] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.996] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:24.996] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:24.996] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:24.998] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:24.998] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@22a316ad),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:24.998] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:24.999] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:24.999] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:24.999] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:25.002] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:25.002] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:25.193] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:25.193] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:25.193] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:25.193] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.193] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.193] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:25.193] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:25.193] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:25.194] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:25.194] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:25.195] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:25.195] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1f517997] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:25.197] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:25.197] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x1f517997] already closed, dropping
[first] [INFO] [06/14/2012 08:19:25.197] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:25.197] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:25.208] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:25.208] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:25.208] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:25.208] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:25.208] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:25.208] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:25.209] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:25.210] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.210] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:25.210] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7b5c9f9d),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:25.210] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.211] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:25.211] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:25.211] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:25.211] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:25.211] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:25.211] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:25.211] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:25.212] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:25.213] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7df33bb0] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.213] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x7df33bb0] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:25.213] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:25.213] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:25.214] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:25.214] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:25.215] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.292] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:25.292] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:25.293] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:25.293] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:25.293] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:25.293] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6140c973] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:25.294] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x6140c973] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:25.294] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:25.294] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:25.294] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:25.294] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:25.294] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:25.294] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:25.294] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5d4177f3),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:25.294] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.295] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:25.295] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:25.295] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:25.296] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:25.296] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:25.306] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:25.306] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:25.307] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.307] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:25.307] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@488e753c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:25.307] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:25.307] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:25.307] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:25.307] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:25.308] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:25.308] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:25.308] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:25.308] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:25.308] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x00b2db5b] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.308] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:25.308] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x00b2db5b] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:25.308] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:25.308] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:25.308] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:25.308] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.391] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:25.392] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:25.392] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:25.392] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.392] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:25.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:25.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:25.393] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:25.393] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:25.393] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:25.394] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x65f1a3a8] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:25.395] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x65f1a3a8] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:25.395] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:25.395] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:25.395] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:25.406] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:25.406] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:25.406] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:25.406] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:25.406] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:25.406] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:25.406] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:25.407] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:25.407] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:25.408] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:25.408] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x460c42b5] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.409] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:25.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:25.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x460c42b5] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:25.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:25.409] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:25.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:25.409] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:25.409] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:25.409] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:25.410] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:25.411] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x53dffb78] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.411] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x53dffb78] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:25.411] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:25.411] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:25.411] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:25.589] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:25.589] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:25.589] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.590] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:25.590] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:25.590] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.590] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:25.590] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:25.591] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:25.591] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:25.592] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:25.593] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x785bda13] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:25.593] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x785bda13] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:25.593] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:25.594] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:25.594] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:25.604] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:25.605] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:25.605] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:25.605] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:25.606] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:25.606] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:25.607] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:25.609] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.609] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:25.610] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4eabaf8c),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:25.610] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.610] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:25.610] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:25.610] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:25.610] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [DEBUG] [06/14/2012 08:19:25.611] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:25.611] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:25.611] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:25.611] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:25.611] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [ERROR] [06/14/2012 08:19:25.611] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:25.612] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:25.614] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x699238ad] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.614] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:25.614] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x699238ad] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:25.614] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:25.614] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:25.616] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:25.616] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.688] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:25.688] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:25.689] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:25.690] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:25.691] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:25.691] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x10def14f] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:25.691] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x10def14f] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:25.691] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:25.691] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:25.691] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:25.691] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:25.691] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:25.691] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:25.692] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4ce2db0),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:25.692] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.692] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:25.692] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:25.692] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:25.693] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:25.693] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:25.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:25.703] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:25.703] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.703] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:25.703] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@14860315),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:25.703] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.703] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:25.703] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:25.704] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:25.704] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:25.704] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:25.704] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:25.704] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:25.704] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6a346239] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.704] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x6a346239] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:25.704] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:25.705] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:25.705] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:25.705] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:25.705] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.787] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:25.787] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:25.788] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:25.788] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.788] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:25.788] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:25.788] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:25.788] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:25.789] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:25.789] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:25.790] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45e881b6] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:25.790] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x45e881b6] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:25.790] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:25.790] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:25.790] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:25.802] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:25.802] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:25.802] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:25.802] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:25.802] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:25.802] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:25.803] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:25.803] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:25.803] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:25.804] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:25.804] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x70ca8f43] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.805] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:25.805] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x70ca8f43] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:25.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:25.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:25.805] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:25.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:25.805] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:25.805] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:25.806] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:25.806] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:25.807] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x07301061] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.807] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x07301061] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:25.807] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:25.807] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:25.807] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:25.985] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:25.986] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:25.986] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:25.987] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:25.988] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:25.989] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@28404eb2),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:25.989] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:25.989] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:25.989] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:25.990] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:25.990] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:25.990] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:25.990] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:25.993] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:25.993] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.000] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:26.000] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.000] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.000] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:26.000] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.000] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.000] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:26.001] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:26.001] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:26.001] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.002] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x13e9d0cc] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.003] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x13e9d0cc] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:26.003] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:26.003] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:26.003] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.018] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:26.018] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:26.018] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.018] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:26.099] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.099] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.099] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.099] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.099] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@65559ba8),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:26.099] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:26.100] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.100] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:26.100] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:26.100] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:26.100] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:26.100] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:26.100] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:26.100] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x575e6691] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.101] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x575e6691] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:26.101] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:26.101] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:26.101] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.101] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:26.101] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.117] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:26.117] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:26.118] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.118] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:26.118] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.118] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x566d0085] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.118] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x566d0085] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:26.119] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:26.119] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:26.119] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:26.119] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.119] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:26.119] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.119] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6a39d072),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:26.119] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.120] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:26.120] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:26.120] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:26.121] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:26.121] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.198] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:26.198] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.198] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:26.198] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.198] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.199] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.199] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.200] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.200] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:26.200] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.200] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@32499f7d),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:26.200] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.200] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:26.200] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:26.201] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:26.201] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:26.201] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:26.201] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:26.202] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.203] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7f85e155] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.203] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x7f85e155] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:26.203] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:26.203] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:26.203] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.203] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:26.204] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.215] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:26.216] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:26.216] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:26.216] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.216] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.216] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:26.216] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:26.216] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:26.216] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:26.217] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.217] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:26.218] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x74d01311] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.218] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:26.218] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x74d01311] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:26.218] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:26.218] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:26.218] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:26.218] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.218] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:26.219] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:26.219] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.219] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:26.220] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1c6250d2] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.220] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x1c6250d2] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:26.220] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:26.220] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:26.220] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:26.395] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:26.396] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.396] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:26.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.396] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:26.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:26.397] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:26.397] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.398] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3cb7cee4] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.398] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3cb7cee4] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:26.398] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:26.398] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:26.398] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.414] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:26.414] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:26.414] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:26.414] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.414] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.414] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:26.414] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:26.414] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:26.415] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:26.415] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.415] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:26.416] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x512fb063] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.416] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x512fb063] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:26.416] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:26.417] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:26.417] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:26.495] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.495] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.496] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.496] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.496] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4c6b3fc7),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:26.497] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:26.497] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.497] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:26.498] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:26.498] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.498] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:26.498] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:26.498] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.498] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x15284903] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.499] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:26.499] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x15284903] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:26.499] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:26.499] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.499] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:26.499] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.513] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:26.514] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:26.514] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.514] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:26.515] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.515] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3bd48043] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.515] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:26.515] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x3bd48043] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:26.515] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:26.515] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:26.515] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.516] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:26.516] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.516] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@424c2849),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:26.516] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.517] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:26.517] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:26.517] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:26.518] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:26.518] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.594] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:26.594] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.595] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.595] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:26.595] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.595] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:26.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:26.597] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:26.597] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.599] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x328b1323] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.599] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:26.600] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x328b1323] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:26.600] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:26.600] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.612] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:26.613] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:26.613] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:26.613] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.613] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:26.613] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.614] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:26.615] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:26.617] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.617] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@58c3d9ac),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:26.617] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.617] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:26.618] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:26.618] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:26.623] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:26.623] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.792] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:26.793] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.793] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:26.794] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.794] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.794] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.794] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:26.795] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:26.795] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:26.796] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.798] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0c481958] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.798] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:26.798] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x0c481958] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:26.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:26.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:26.798] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.799] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:26.799] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:26.799] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:26.799] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:26.800] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.801] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x082db73e] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.802] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x082db73e] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:26.802] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:26.802] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:26.802] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.810] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:26.810] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:26.810] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:26.810] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.810] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.810] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:26.811] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:26.811] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:26.812] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.812] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1dd61ee4),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:26.812] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.812] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:26.813] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:26.813] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:26.815] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:26.816] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.891] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.892] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.892] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.892] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.893] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:26.893] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4acf7fd0),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:26.893] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.893] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:26.894] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.894] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:26.894] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:26.894] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:26.894] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:26.894] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7cd0570e] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.894] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x7cd0570e] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:26.895] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:26.895] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:26.895] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.896] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:26.896] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:26.910] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:26.910] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:26.911] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.911] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:26.912] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.912] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x54861a09] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.912] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:26.912] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x54861a09] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:26.912] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:26.912] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:26.912] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.913] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:26.913] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:26.913] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5576b9ea),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:26.913] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.914] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:26.914] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:26.914] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:26.915] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:26.916] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.989] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:26.990] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:26.990] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:26.990] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:26.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.991] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.991] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:26.991] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:26.992] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@27905a42),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:26.992] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.992] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:26.992] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:26.992] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:26.992] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:26.992] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:26.992] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:26.993] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:26.994] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7615b758] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:26.994] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x7615b758] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:26.994] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:26.994] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:26.994] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:26.995] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:26.995] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.008] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:27.008] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:27.008] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.008] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:27.008] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.008] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:27.008] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:27.009] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:27.009] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.010] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@620b5b80),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:27.010] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.010] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:27.010] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:27.010] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:27.014] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:27.014] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:27.188] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:27.188] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:27.188] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:27.188] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:27.189] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:27.189] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.189] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.190] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.191] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:27.191] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.191] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@24dbf79d),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:27.191] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:27.191] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.191] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:27.192] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:27.192] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:27.192] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:27.192] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:27.193] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.194] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x655d7752] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.194] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x655d7752] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:27.194] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:27.194] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:27.194] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:27.195] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:27.195] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.195] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:27.195] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:27.195] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.196] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:27.196] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:27.196] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:27.197] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6040f3c1] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.197] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6040f3c1] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:27.197] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:27.197] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:27.198] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:27.206] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:27.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:27.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:27.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.206] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.207] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:27.207] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:27.207] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:27.208] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.208] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2c154024),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:27.208] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.208] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:27.209] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:27.209] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:27.211] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:27.212] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:27.288] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.288] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.289] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.289] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.289] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:27.289] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@669d2f26),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:27.290] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.290] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:27.290] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.290] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:27.290] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:27.291] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:27.291] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:27.291] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x59829c6b] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.291] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x59829c6b] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:27.292] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:27.292] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:27.292] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:27.292] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:27.292] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.305] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:27.305] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:27.306] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.306] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:27.306] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.306] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x27261218] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.306] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x27261218] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:27.307] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:27.307] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:27.307] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:27.307] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:27.307] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:27.307] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.308] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3b7c680),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:27.308] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.308] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:27.308] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:27.309] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:27.309] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:27.310] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:27.386] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:27.386] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:27.387] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:27.387] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:27.387] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:27.387] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.388] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.390] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.390] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:27.390] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.391] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@132c08c7),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:27.391] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.391] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:27.391] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:27.391] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:27.392] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:27.392] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:27.392] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:27.393] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.395] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2ff44aa6] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.395] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2ff44aa6] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:27.395] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:27.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:27.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:27.396] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:27.397] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.404] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:27.404] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:27.404] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.404] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:27.404] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.405] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:27.405] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:27.405] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:27.405] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:27.405] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.406] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:27.406] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x08746d23] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.406] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x08746d23] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:27.407] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:27.407] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:27.407] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:27.584] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:27.585] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:27.585] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:27.585] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:27.586] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:27.586] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.587] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.588] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.588] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:27.589] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.589] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1830e4a7),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:27.589] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.589] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:27.589] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:27.590] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:27.590] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:27.590] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:27.591] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:27.592] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.593] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x31f39c59] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.593] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x31f39c59] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:27.593] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:27.593] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:27.593] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:27.595] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:27.595] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.602] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:27.602] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:27.603] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:27.603] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.603] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.603] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:27.603] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:27.603] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:27.604] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:27.604] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.604] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:27.606] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x409a44d6] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.606] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:27.606] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x409a44d6] already closed, dropping
[first] [INFO] [06/14/2012 08:19:27.606] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:27.606] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:27.683] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.683] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.683] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.684] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.684] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:27.684] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6a7be687),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:27.684] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.685] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:27.685] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.685] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:27.685] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:27.685] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:27.685] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:27.686] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6c5bdfae] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.686] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x6c5bdfae] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:27.686] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:27.686] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:27.687] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:27.687] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:27.687] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.701] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:27.701] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:27.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:27.702] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x49bfd53d] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.702] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x49bfd53d] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:27.702] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:27.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:27.702] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:27.703] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:27.703] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:27.703] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.703] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@40d0726d),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:27.703] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.704] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:27.704] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:27.704] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:27.705] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:27.705] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:27.781] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:27.782] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:27.782] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:27.782] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:27.783] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:27.783] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.784] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.785] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.785] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:27.785] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.786] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@123aabc1),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:27.786] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.786] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:27.786] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:27.786] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [DEBUG] [06/14/2012 08:19:27.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:27.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [ERROR] [06/14/2012 08:19:27.787] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:27.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [ERROR] [06/14/2012 08:19:27.787] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:27.788] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.789] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x19e142a5] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.789] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:27.790] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x19e142a5] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:27.790] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:27.790] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:27.791] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:27.792] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.800] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:27.800] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:27.800] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:27.800] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.801] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.801] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:27.801] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:27.801] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:27.801] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:27.801] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.802] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:27.802] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5a7691c0] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.802] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:27.803] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x5a7691c0] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:27.803] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:27.803] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:27.803] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:27.803] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:27.803] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:27.803] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:27.803] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:27.804] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:27.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x73f17a73] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:27.805] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:27.805] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x73f17a73] already closed, dropping
[first] [INFO] [06/14/2012 08:19:27.805] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:27.805] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:27.979] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:27.980] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:27.981] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:27.981] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:27.981] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:27.981] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:27.981] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:27.982] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:27.982] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:27.983] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.985] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x36029483] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.985] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:27.985] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x36029483] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:27.985] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:27.985] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [INFO] [06/14/2012 08:19:27.985] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:27.986] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:27.986] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:27.986] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:27.986] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:27.987] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:27.988] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1b5268d7] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:27.988] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x1b5268d7] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:27.989] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:27.989] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:27.989] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:27.998] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:27.998] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:27.999] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:27.999] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.999] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:27.999] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:27.999] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:27.999] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.000] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.000] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7896b1b8),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:28.000] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.000] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:28.001] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.001] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:28.004] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:28.004] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:28.080] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:28.080] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:28.080] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.081] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:28.081] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:28.081] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1652d654),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.081] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.082] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:28.082] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:28.082] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.082] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.083] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:28.083] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:28.083] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1875303f] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.083] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x1875303f] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:28.083] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:28.083] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:28.084] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:28.084] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:28.084] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.097] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:28.097] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:28.098] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:28.098] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:28.098] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:28.098] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6d6de4e1] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.098] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x6d6de4e1] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:28.099] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:28.099] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:28.099] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:28.099] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:28.099] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.099] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.099] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6dc8f3cd),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:28.099] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.100] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:28.100] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.100] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:28.101] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:28.101] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:28.178] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:28.178] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:28.178] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:28.178] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:28.178] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:28.178] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:28.178] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:28.179] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:28.179] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:28.179] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:28.180] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1005354a] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.180] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x1005354a] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:28.180] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:28.180] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:28.180] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:28.196] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:28.196] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:28.196] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:28.196] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.196] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.196] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:28.196] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:28.197] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.198] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.198] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3f0dbef1),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:28.198] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.198] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:28.199] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:28.199] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.199] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:28.199] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.199] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:28.201] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:28.201] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:28.375] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:28.376] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:28.376] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:28.376] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:28.377] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.377] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:28.377] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:28.377] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@549b6976),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.377] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.378] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:28.378] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:28.378] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:28.378] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:28.378] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:28.378] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:28.381] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.381] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:28.381] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2728ed5f] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.381] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:28.382] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:28.382] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2728ed5f] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:28.382] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:28.382] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:28.394] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:28.394] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:28.394] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:28.394] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.394] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.394] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:28.394] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:28.394] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.395] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.395] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@53b4b24d),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:28.395] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.396] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:28.396] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.396] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:28.398] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:28.399] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:28.409] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:28.409] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:28.409] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.494] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:28.494] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:28.494] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:28.494] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:28.494] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:28.495] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7c260829] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.495] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:28.495] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x7c260829] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:28.495] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:28.495] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:28.495] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:28.495] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.495] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.495] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7122e0b3),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:28.495] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.496] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:28.496] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.496] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:28.497] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:28.497] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:28.508] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:28.508] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:28.508] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.508] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:28.508] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:28.508] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@492d0430),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.508] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.509] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:28.509] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:28.509] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.509] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:28.509] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:28.509] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:28.509] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2a6e10d8] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.509] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x2a6e10d8] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:28.509] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:28.509] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:28.509] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:28.510] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:28.510] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.592] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:28.592] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:28.592] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.592] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:28.592] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:28.592] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.593] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:28.593] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.594] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.594] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2cdf88d2),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:28.594] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.594] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:28.595] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.595] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:28.598] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:28.598] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:28.607] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:28.607] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:28.607] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:28.607] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:28.608] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:28.608] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:28.608] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:28.609] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.609] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@77c7bee2),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.609] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.610] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:28.610] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:28.610] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:28.612] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:28.612] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.790] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:28.790] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:28.791] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:28.791] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.791] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.791] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:28.791] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:28.791] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:28.793] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:28.793] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:28.793] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.795] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3c250cce] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.795] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3c250cce] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:28.795] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:28.795] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:28.796] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:28.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:28.805] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:28.805] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:28.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:28.805] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:28.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:28.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:28.806] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:28.806] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:28.806] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:28.807] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4aa0560e] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.807] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x4aa0560e] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:28.807] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:28.807] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:28.807] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:28.889] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:28.889] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:28.891] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:28.891] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:28.891] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:28.891] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x088d00c6] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.892] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x088d00c6] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:28.892] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:28.892] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:28.892] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:28.892] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:28.892] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.893] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.893] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@659c2931),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:28.893] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.893] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:28.894] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.894] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:28.895] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:28.895] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:28.903] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:28.904] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:28.904] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.904] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:28.904] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:28.904] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@733638d4),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.904] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.905] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:28.905] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:28.905] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:28.905] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:28.905] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.905] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:28.906] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6ccd2163] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:28.906] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:28.906] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x6ccd2163] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:28.906] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:28.906] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:28.907] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:28.907] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.988] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:28.989] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:28.989] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:28.989] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.990] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:28.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:28.990] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:28.990] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:28.991] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:28.991] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:28.992] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:28.994] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2bd9c3e7] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:28.994] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2bd9c3e7] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:28.994] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:28.994] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:28.994] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:29.003] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:29.003] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:29.003] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:29.003] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.003] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.004] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.004] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:29.004] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:29.004] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:29.005] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:29.006] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x13a66c87] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.006] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x13a66c87] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:29.006] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:29.006] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:29.006] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:29.186] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:29.186] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:29.186] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:29.186] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:29.187] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:29.187] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:29.187] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:29.187] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:29.188] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:29.188] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:29.188] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:29.189] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x21044daf] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:29.190] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:29.190] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x21044daf] already closed, dropping
[first] [INFO] [06/14/2012 08:19:29.190] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:29.190] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:29.201] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:29.201] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:29.201] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:29.201] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.201] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.201] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.201] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:29.202] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:29.202] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:29.203] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:29.203] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x550a6723] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.204] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:29.204] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x550a6723] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:29.204] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:29.204] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:29.299] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.300] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:29.300] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.300] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.300] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1d6535bf),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:29.300] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.301] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:29.301] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:29.301] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:29.301] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:29.301] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:29.301] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:29.302] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:29.302] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7bfcb845] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.302] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x7bfcb845] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:29.302] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:29.302] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:29.302] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:29.302] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:29.303] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:29.318] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:29.318] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:29.319] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:29.319] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:29.319] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:29.319] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x72d86c58] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:29.319] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:29.319] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x72d86c58] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:29.319] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:29.319] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:29.319] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:29.319] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:29.320] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:29.320] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@616181be),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:29.320] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.320] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:29.320] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:29.321] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:29.321] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:29.321] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:29.398] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:29.399] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:29.399] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:29.399] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.399] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.399] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.399] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:29.400] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:29.400] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:29.400] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:29.401] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3f1de914] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:29.401] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3f1de914] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:29.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:29.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:29.417] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:29.417] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:29.417] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:29.418] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:29.418] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:29.418] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:29.418] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:29.418] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:29.418] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:29.418] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:29.419] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:29.419] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x57398044] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:29.420] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x57398044] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:29.420] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:29.420] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:29.420] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:29.596] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:29.597] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:29.598] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:29.598] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.598] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.598] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.599] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:29.601] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.601] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:29.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@237aaeec),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:29.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.601] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.602] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:29.602] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:29.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:29.603] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:29.603] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:29.603] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:29.604] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:29.605] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4b721a26] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.605] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:29.605] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x4b721a26] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:29.605] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:29.606] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:29.607] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:29.607] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:29.615] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:29.615] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:29.615] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:29.615] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:29.615] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:29.616] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:29.616] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:29.616] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:29.617] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:29.617] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4bcd2d49),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:29.617] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.618] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:29.618] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:29.618] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:29.620] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:29.621] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:29.696] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.696] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:29.697] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.697] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.697] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5d11c3f0),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:29.697] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:29.697] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:29.697] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:29.697] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:29.698] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:29.698] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:29.698] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:29.698] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:29.698] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3956f14c] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.698] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x3956f14c] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:29.698] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:29.698] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:29.698] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:29.698] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:29.698] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:29.714] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:29.714] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:29.714] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:29.714] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:29.714] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:29.715] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0b583a80] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:29.715] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:29.715] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x0b583a80] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:29.715] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:29.715] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:29.715] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:29.715] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:29.715] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:29.715] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@d2a7c1e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:29.715] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.715] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:29.716] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:29.716] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:29.716] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:29.716] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:29.795] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:29.796] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:29.796] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:29.796] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.796] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.796] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.797] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:29.797] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:29.798] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:29.798] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:29.800] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1c6250d2] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.800] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:29.800] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x1c6250d2] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:29.800] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:29.800] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:29.800] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.800] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:29.800] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:29.801] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:29.801] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:29.802] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:29.803] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x21a722ef] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.803] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x21a722ef] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:29.803] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:29.803] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:29.803] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:29.813] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:29.814] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:29.814] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:29.814] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:29.814] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:29.814] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:29.814] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:29.815] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:29.816] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:29.816] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2a788315),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:29.816] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.816] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:29.817] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:29.817] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:29.820] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:29.820] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:29.992] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:29.992] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:29.992] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:29.992] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.993] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:29.993] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.993] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:29.994] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.994] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:29.994] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@404fe94c),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:29.994] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.995] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:29.995] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:29.995] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:29.995] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:29.995] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:29.995] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:29.995] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:29.996] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:29.997] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3e5d9d90] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:29.997] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3e5d9d90] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:29.997] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:29.997] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:29.997] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:29.998] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:29.998] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.010] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:30.011] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:30.011] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.011] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:30.011] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.011] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:30.011] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:30.011] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:30.012] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:30.012] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.012] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:30.013] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2c538cc2] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.013] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:30.013] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2c538cc2] already closed, dropping
[first] [INFO] [06/14/2012 08:19:30.013] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:30.013] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:30.092] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.092] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.093] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.093] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.093] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:30.094] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@705789f3),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:30.094] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.094] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:30.095] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:30.095] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:30.095] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:30.095] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.095] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:30.095] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0321ea24] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.096] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x0321ea24] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:30.096] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:30.096] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:30.096] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:30.096] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:30.096] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.110] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:30.110] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:30.111] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.111] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:30.112] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.112] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5113de03] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.112] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x5113de03] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:30.112] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:30.112] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:30.112] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:30.113] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:30.113] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:30.113] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.113] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@558fee4f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:30.113] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.114] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:30.114] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:30.114] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:30.115] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:30.115] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:30.191] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:30.191] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:30.191] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:30.191] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:30.192] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:30.192] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.192] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.193] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.194] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:30.194] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.194] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6629a046),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:30.194] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.194] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:30.194] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:30.195] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [DEBUG] [06/14/2012 08:19:30.195] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:30.195] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.195] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [ERROR] [06/14/2012 08:19:30.195] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.195] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:30.195] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:30.196] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.197] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x36d98810] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.197] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:30.197] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x36d98810] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:30.197] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:30.198] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:30.199] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:30.199] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.209] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:30.209] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:30.209] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.209] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:30.209] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.210] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:30.210] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:30.210] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:30.211] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.211] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:30.211] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:30.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@49e61582),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:30.211] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.211] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:30.211] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:30.211] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:30.212] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [ERROR] [06/14/2012 08:19:30.212] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:30.212] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [ERROR] [06/14/2012 08:19:30.212] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:30.212] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:30.212] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [DEBUG] [06/14/2012 08:19:30.212] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:30.213] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x208e2fb5] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.213] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:30.213] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x208e2fb5] already closed, dropping
[first] [INFO] [06/14/2012 08:19:30.213] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:30.213] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:30.215] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:30.215] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:30.388] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:30.389] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:30.389] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:30.389] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:30.389] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:30.389] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.390] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.391] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.391] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:30.391] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1c8825a5),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:30.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.392] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:30.392] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:30.392] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:30.393] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:30.393] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:30.393] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:30.394] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.395] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e5f8245] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.395] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2e5f8245] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:30.395] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:30.395] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:30.395] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:30.396] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:30.396] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.396] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:30.396] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:30.396] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.397] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:30.397] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:30.397] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:30.398] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2e893a4a] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.398] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2e893a4a] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:30.398] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:30.398] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:30.398] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:30.407] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:30.407] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:30.407] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:30.407] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.407] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:30.407] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.407] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:30.407] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:30.408] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:30.408] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.408] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:30.409] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5dc37b8f] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.409] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:30.409] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x5dc37b8f] already closed, dropping
[first] [INFO] [06/14/2012 08:19:30.409] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:30.409] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:30.488] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.488] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.488] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.488] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.488] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@36c51089),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:30.489] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:30.489] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.489] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:30.489] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.489] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:30.489] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:30.489] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.489] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:30.489] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43c0ae76] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.490] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x43c0ae76] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:30.490] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:30.490] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:30.490] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:30.490] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:30.490] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.506] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:30.506] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:30.507] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.507] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:30.507] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.507] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x5ce345c2] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.507] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:30.507] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x5ce345c2] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:30.507] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:30.507] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:30.507] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:30.508] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:30.508] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.508] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@35e5ebbf),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:30.508] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.508] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:30.508] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:30.508] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:30.509] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:30.509] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:30.586] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:30.586] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:30.587] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:30.587] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:30.587] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:30.587] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.587] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:30.587] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:30.587] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:30.588] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.588] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3ad6a0e0] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.588] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3ad6a0e0] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:30.589] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:30.589] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:30.589] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:30.605] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:30.605] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:30.605] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:30.605] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.605] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:30.605] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.605] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:30.605] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:30.606] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:30.606] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.608] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:30.609] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x788bf135] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.609] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x788bf135] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:30.609] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:30.609] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:30.609] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:30.785] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:30.785] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:30.785] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:30.785] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:30.785] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:30.785] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.785] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:30.786] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:30.786] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:30.786] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.787] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3cecfaea] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.787] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:30.787] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3cecfaea] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:30.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:30.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:30.787] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:30.787] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:30.788] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:30.788] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:30.788] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.789] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x26f44031] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.789] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x26f44031] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:30.789] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:30.789] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:30.789] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:30.803] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:30.803] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:30.803] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:30.803] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.803] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.803] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:30.803] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:30.803] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:30.804] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:30.804] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.804] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:30.805] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7aa30a4e] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.805] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x7aa30a4e] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:30.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:30.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:30.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:30.884] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.884] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.885] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.885] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.885] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@8aee908),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:30.885] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.885] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:30.885] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:30.886] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:30.886] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.886] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:30.886] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:30.886] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:30.886] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x134510ac] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.886] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x134510ac] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:30.886] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:30.886] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:30.886] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:30.886] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:30.886] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:30.902] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:30.902] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:30.903] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.903] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:30.903] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:30.903] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x06d4b473] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.903] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x06d4b473] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:30.903] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:30.903] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:30.903] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:30.904] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:30.904] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:30.904] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:30.904] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1ea87e7b),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:30.904] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.904] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:30.904] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:30.905] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:30.905] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:30.905] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:30.983] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:30.983] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:30.983] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:30.983] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:30.983] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:30.983] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:30.983] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:30.984] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:30.984] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:30.984] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.985] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6dc98c1b] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.985] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:30.985] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:30.985] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6dc98c1b] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:30.985] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [INFO] [06/14/2012 08:19:30.985] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:30.985] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:30.985] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:30.986] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:30.986] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:30.986] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:30.987] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0b2929b2] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:30.987] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x0b2929b2] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:30.987] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:30.987] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:30.987] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:31.001] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:31.001] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:31.001] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:31.001] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.001] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.001] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:31.001] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:31.001] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:31.002] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:31.002] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:31.002] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.003] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x52e5376a] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.003] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x52e5376a] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:31.003] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:31.003] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:31.003] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:31.180] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:31.180] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:31.181] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:31.181] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:31.181] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:31.181] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.181] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:31.182] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.182] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:31.182] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@61f873dd),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:31.182] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.182] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.182] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:31.182] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:31.183] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:31.183] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:31.183] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:31.183] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:31.183] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:31.184] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7e7ee46a] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.184] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x7e7ee46a] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:31.184] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:31.184] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:31.184] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:31.186] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:31.186] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.199] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:31.199] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:31.199] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:31.199] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.199] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.200] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:31.200] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:31.200] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.200] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.201] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@47ffccd6),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:31.201] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.201] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:31.201] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:31.201] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:31.203] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:31.204] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:31.280] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.281] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:31.281] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.281] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.282] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6fd33eef),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:31.282] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.282] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:31.282] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:31.283] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:31.283] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:31.283] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.283] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:31.283] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:31.283] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x15c8f644] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.283] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x15c8f644] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:31.284] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:31.284] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:31.284] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:31.284] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:31.284] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.298] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:31.298] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:31.299] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:31.299] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:31.299] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:31.300] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6ae11a87] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.300] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x6ae11a87] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:31.300] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:31.300] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:31.300] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:31.300] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:31.300] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.301] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.301] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@590e130c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:31.301] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.301] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:31.301] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:31.302] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:31.302] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:31.303] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:31.378] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:31.379] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:31.379] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:31.379] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:31.379] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:31.379] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.379] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:31.380] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.380] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:31.380] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6f9702b1),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:31.380] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.380] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.380] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:31.380] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:31.381] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:31.381] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:31.381] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:31.381] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:31.382] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:31.382] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2331ebb3] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.382] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2331ebb3] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:31.382] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:31.382] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:31.382] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:31.382] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.382] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:31.383] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:31.383] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:31.383] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:31.383] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.383] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:31.384] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:31.384] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3103074e] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.384] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3103074e] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:31.384] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:31.384] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:31.384] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:31.398] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:31.398] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:31.398] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:31.398] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.398] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.398] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:31.398] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:31.398] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:31.399] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:31.399] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:31.399] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.400] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x52988dd6] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.400] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:31.400] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x52988dd6] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:31.400] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:31.400] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:31.400] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:31.400] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:31.400] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:31.401] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:31.401] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:31.401] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.402] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3e3e3c83] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.402] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3e3e3c83] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:31.402] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:31.402] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:31.402] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:31.576] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:31.577] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:31.577] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:31.577] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:31.577] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:31.577] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.578] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:31.579] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.579] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:31.579] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.579] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2677622b),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:31.579] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:31.580] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.580] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:31.580] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:31.580] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:31.580] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:31.581] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:31.581] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:31.582] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x67ce08c7] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.582] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x67ce08c7] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:31.582] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:31.583] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:31.583] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:31.583] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.583] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:31.583] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:31.584] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:31.584] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:31.585] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.585] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:31.586] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x404b7c69] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.586] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:31.586] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x404b7c69] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:31.586] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:31.586] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:31.586] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:31.595] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:31.595] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:31.595] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:31.595] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:31.595] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.595] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.595] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:31.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.597] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.597] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@65b1fd9c),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:31.597] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.597] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:31.597] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:31.598] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:31.600] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:31.600] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:31.694] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:31.694] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:31.695] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:31.695] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:31.695] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:31.695] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x088140ed] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.695] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:31.695] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x088140ed] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:31.695] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:31.695] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:31.695] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:31.695] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.696] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.696] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1cf15b84),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:31.696] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.696] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:31.696] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:31.697] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:31.697] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:31.697] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:31.709] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.709] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:31.709] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.709] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.709] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@777490ad),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:31.709] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.709] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:31.710] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:31.710] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:31.710] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:31.710] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:31.710] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:31.710] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:31.710] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x05c68b20] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.710] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x05c68b20] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:31.710] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:31.711] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:31.711] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:31.711] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:31.711] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.793] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:31.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:31.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:31.794] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.794] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:31.794] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.794] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:31.794] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:31.794] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:31.794] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:31.795] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.796] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x397d812b] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.796] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:31.796] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:31.796] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x397d812b] already closed, dropping
[first] [INFO] [06/14/2012 08:19:31.796] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:31.796] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:31.796] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:31.796] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:31.796] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:31.796] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:31.797] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.797] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2ced55ec] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.797] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2ced55ec] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:31.797] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:31.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:31.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:31.807] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:31.808] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:31.808] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:31.808] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:31.808] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:31.808] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.808] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:31.809] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:31.809] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:31.809] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:31.810] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6d6de4e1] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.810] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:31.810] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:31.810] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6d6de4e1] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:31.810] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:31.810] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:31.810] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:31.810] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:31.810] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:31.810] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:31.811] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:31.811] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x79b0edb2] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.811] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x79b0edb2] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:31.811] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:31.811] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:31.811] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:31.991] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:31.991] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:31.992] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:31.992] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.992] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:31.992] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:31.992] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:31.993] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:31.994] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:31.995] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@21e8bf76),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:31.995] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:31.995] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:31.996] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:31.996] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:32.000] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:32.000] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.005] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:32.006] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.006] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.006] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.006] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.006] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.006] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:32.007] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:32.007] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:32.008] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.008] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x34bdb859] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.008] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:32.008] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x34bdb859] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:32.009] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:32.009] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:32.009] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:32.009] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.009] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:32.009] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:32.010] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:32.010] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.011] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x36a3bb2e] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.011] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x36a3bb2e] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:32.011] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:32.011] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:32.011] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:32.090] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:32.090] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:32.092] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.092] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:32.092] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.092] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3771ed5e] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.093] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:32.093] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x3771ed5e] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:32.093] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:32.093] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:32.093] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:32.093] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:32.093] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.094] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1d10caf0),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:32.094] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.094] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:32.095] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.095] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:32.096] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:32.096] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.105] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.106] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.106] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.106] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.106] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@145a25f3),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.106] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:32.106] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.107] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:32.107] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:32.107] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.107] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:32.107] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:32.107] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:32.108] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2a717ef5] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.108] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x2a717ef5] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:32.108] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:32.108] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:32.108] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:32.108] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:32.108] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.189] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:32.189] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:32.190] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:32.190] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.190] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:32.190] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.190] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:32.190] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:32.191] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.191] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:32.191] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:32.191] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@774943d6),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:32.191] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.191] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:32.191] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:32.192] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.192] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:32.192] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:32.192] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.192] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:32.193] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6b04d3c8] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.193] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6b04d3c8] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:32.193] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:32.194] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:32.194] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:32.194] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:32.194] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.204] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:32.204] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.204] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.204] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.204] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.204] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.204] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.205] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.205] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.205] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.205] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@604c9c17),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.205] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.205] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:32.205] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:32.206] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:32.206] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:32.206] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:32.206] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:32.206] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.207] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x459bdb65] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.207] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.207] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x459bdb65] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:32.207] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:32.207] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:32.208] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:32.209] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.387] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:32.387] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:32.387] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:32.388] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:32.388] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.388] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.388] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:32.389] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:32.390] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.390] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:32.390] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:32.390] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7d05e560),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:32.390] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.390] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:32.390] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:32.391] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [ERROR] [06/14/2012 08:19:32.391] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:32.391] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:32.391] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.391] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:32.392] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0e1641c0] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.392] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x0e1641c0] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:32.393] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:32.393] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:32.393] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:32.394] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:32.394] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.401] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:32.402] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.402] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.402] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.402] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.402] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.402] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.403] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.403] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.403] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.403] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@38da9246),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.403] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.403] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:32.403] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:32.403] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:32.403] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:32.403] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:32.404] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:32.404] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.405] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x38e609c9] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.405] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x38e609c9] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:32.405] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:32.405] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:32.405] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:32.406] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:32.406] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.486] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:32.486] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:32.486] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.486] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:32.486] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.487] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x38b72ce1] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.487] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:32.487] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:32.487] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x38b72ce1] already closed, dropping
[first] [INFO] [06/14/2012 08:19:32.487] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:32.487] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:32.487] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:32.487] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.487] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3761f888),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:32.487] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.488] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:32.488] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.488] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:32.489] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:32.489] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.501] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.501] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.501] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.501] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.501] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@780324ff),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.501] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:32.502] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.502] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:32.502] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.502] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.502] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:32.502] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:32.502] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:32.502] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x16721ee7] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.502] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.502] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x16721ee7] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:32.503] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:32.503] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:32.503] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:32.503] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.585] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:32.585] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.585] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:32.586] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:32.587] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.587] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@777d57d6),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:32.588] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.588] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:32.588] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:32.588] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.588] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:32.588] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.588] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:32.590] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:32.591] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.599] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:32.599] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.600] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.600] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.600] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.600] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.600] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.601] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.601] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:32.601] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.601] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5fa6fb3e),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.601] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.601] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:32.601] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:32.601] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [DEBUG] [06/14/2012 08:19:32.601] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:32.601] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.602] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [ERROR] [06/14/2012 08:19:32.602] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.602] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:32.602] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:32.602] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.602] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4bb8d481] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.603] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x4bb8d481] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:32.603] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:32.603] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:32.603] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:32.603] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.603] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:32.603] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:32.603] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:32.603] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:32.604] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.604] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2cccbab7] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.604] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2cccbab7] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:32.604] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:32.604] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:32.604] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:32.605] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:32.605] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.618] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:32.618] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:32.618] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.618] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[fourth] [DEBUG] [06/14/2012 08:19:32.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:32.798] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.798] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.798] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.798] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.799] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.799] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.799] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.799] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.799] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2400218d),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.799] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.800] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:32.800] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:32.800] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:32.800] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:32.800] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:32.800] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:32.801] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.801] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x049ac272] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.801] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.801] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x049ac272] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:32.802] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:32.802] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:32.803] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:32.803] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.815] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:32.816] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:32.816] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:32.816] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.816] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.816] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:32.816] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:32.816] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:32.817] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.817] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@15128ee5),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:32.817] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.817] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:32.818] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.818] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:32.820] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:32.820] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.897] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.897] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.897] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.897] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.897] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:32.897] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7f8a8bfa),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.897] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.897] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:32.898] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:32.898] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:32.898] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:32.898] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.898] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:32.898] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x60ec0b80] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.898] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.898] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x60ec0b80] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:32.898] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:32.898] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:32.899] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:32.899] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:32.915] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:32.915] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:32.916] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.916] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:32.916] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:32.916] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x75636731] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.916] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x75636731] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:32.916] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:32.916] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:32.916] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:32.916] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:32.916] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:32.917] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:32.917] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2e29d50d),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:32.917] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.917] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:32.917] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.917] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:32.918] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:32.918] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.996] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:32.996] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:32.997] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:32.997] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.997] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:32.997] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.998] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:32.999] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:32.999] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:32.999] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:32.999] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4335236e),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:32.999] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:32.999] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.000] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:33.000] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:33.000] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.000] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:33.001] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:33.001] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:33.002] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x55d866c5] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.002] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x55d866c5] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:33.003] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:33.003] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:33.003] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:33.003] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:33.003] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.003] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:33.004] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:33.004] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.005] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:33.005] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:33.005] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:33.006] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x76cc518c] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.006] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:33.006] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x76cc518c] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:33.006] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:33.006] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:33.014] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:33.014] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:33.014] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:33.014] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:33.014] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.014] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.014] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:33.014] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:33.015] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:33.015] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.016] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:33.016] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0bc6f1f4] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.016] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x0bc6f1f4] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:33.016] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:33.016] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:33.017] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:33.193] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:33.194] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:33.194] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:33.194] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:33.194] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:33.194] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.195] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:33.195] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.195] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:33.195] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.196] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5b09062e),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:33.196] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.196] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:33.196] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:33.196] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:33.197] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.197] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:33.197] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:33.198] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:33.198] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2d74e4b3] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.198] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2d74e4b3] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:33.199] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:33.199] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:33.199] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:33.200] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:33.201] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.212] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:33.212] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:33.212] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:33.212] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.212] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.212] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:33.212] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:33.212] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:33.213] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:33.213] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.213] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:33.214] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x526a4268] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.214] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x526a4268] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:33.214] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:33.214] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:33.214] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:33.294] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.294] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:33.294] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.295] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.295] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6102d81c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:33.295] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.295] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:33.296] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:33.296] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:33.296] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:33.296] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:33.296] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:33.296] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:33.297] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x01ba4806] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.297] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:33.297] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x01ba4806] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:33.297] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:33.297] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:33.298] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:33.298] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.311] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:33.311] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:33.312] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.312] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:33.312] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.312] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x17cb0a16] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.313] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:33.313] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x17cb0a16] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:33.313] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:33.313] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:33.313] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:33.313] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:33.313] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.314] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@c33f45e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:33.314] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.314] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:33.315] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:33.315] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:33.315] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:33.316] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:33.392] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:33.392] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:33.392] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:33.392] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:33.392] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:33.392] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.393] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:33.393] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:33.393] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:33.394] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:33.394] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x35e5ebbf] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.394] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:33.394] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x35e5ebbf] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:33.394] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:33.394] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:33.410] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:33.411] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:33.411] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:33.411] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.411] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.411] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:33.411] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:33.411] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:33.412] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.412] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1a8fa0f0),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:33.412] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.412] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:33.413] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:33.413] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:33.415] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:33.415] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:33.590] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:33.591] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:33.591] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:33.591] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:33.591] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:33.591] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.591] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:33.593] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:33.596] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:33.597] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:33.597] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3fdb8a73] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.598] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:33.598] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3fdb8a73] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:33.598] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:33.598] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:33.608] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:33.609] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:33.609] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:33.609] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.609] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:33.609] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.609] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:33.610] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:33.611] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.611] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:33.611] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:33.611] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4511e86f),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:33.611] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.611] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:33.611] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:33.612] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:33.612] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:33.612] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:33.612] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.613] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:33.614] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1ab2f2d6] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.614] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x1ab2f2d6] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:33.614] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:33.614] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:33.614] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:33.615] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:33.616] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:33.689] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.689] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:33.689] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.689] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.689] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:33.689] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@32d2bb53),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:33.689] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.689] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:33.690] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:33.690] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:33.690] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:33.690] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.690] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:33.690] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x29086036] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.690] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:33.690] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x29086036] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:33.690] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:33.690] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:33.690] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:33.691] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.707] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:33.707] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:33.708] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.708] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:33.708] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.708] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4eafccbe] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.708] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x4eafccbe] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:33.708] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:33.708] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:33.708] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:33.708] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:33.708] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:33.708] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.709] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@72ebbf5c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:33.709] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.709] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:33.709] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:33.709] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:33.710] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:33.710] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:33.788] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:33.788] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:33.789] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:33.789] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:33.789] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:33.789] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.790] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:33.790] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.791] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@27573872),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:33.791] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.791] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:33.791] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:33.792] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:33.795] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:33.795] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.806] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:33.806] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:33.806] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:33.806] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.806] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:33.806] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:33.806] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:33.807] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:33.807] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.807] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:33.807] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:33.808] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@12ac706a),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:33.808] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.808] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:33.808] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:33.808] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:33.808] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:33.809] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:33.809] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:33.809] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:33.810] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x770848b9] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:33.810] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x770848b9] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:33.810] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:33.810] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:33.810] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:33.811] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:33.812] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:33.986] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:33.986] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:33.987] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:33.987] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:33.987] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:33.987] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.987] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:33.988] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:33.988] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:33.989] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:33.989] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2a134eca] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.990] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:33.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2a134eca] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:33.990] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:33.990] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:33.990] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:33.990] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:33.990] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:33.991] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:33.991] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:33.992] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:33.993] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x15dbac11] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:33.993] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x15dbac11] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:33.993] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:33.993] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:33.993] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.004] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:34.004] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:34.005] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:34.005] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.005] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:34.005] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.005] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.005] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:34.006] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:34.006] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.006] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.007] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x69123787] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.007] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:34.007] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:34.007] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x69123787] already closed, dropping
[first] [INFO] [06/14/2012 08:19:34.007] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:34.007] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.007] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.007] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:34.008] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:34.008] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.009] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.009] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x15c330aa] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.009] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x15c330aa] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:34.009] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:34.009] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:34.009] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:34.085] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.085] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:34.085] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.085] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.085] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:34.085] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@d542094),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:34.085] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.085] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:34.086] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.086] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.086] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:34.086] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.086] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:34.086] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0e26db62] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.086] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:34.086] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x0e26db62] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:34.086] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:34.086] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.086] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:34.087] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.103] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.103] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:34.103] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.103] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:34.103] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.104] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x78aea5cd] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.104] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:34.104] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x78aea5cd] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:34.104] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:34.104] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:34.104] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.104] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.104] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.104] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@11be650f),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:34.104] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.105] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:34.105] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:34.105] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:34.105] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:34.105] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:34.183] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:34.183] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:34.183] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:34.183] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:34.184] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:34.184] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.184] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:34.184] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.184] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:34.184] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.185] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1c898b41),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:34.185] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.185] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:34.185] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:34.185] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:34.185] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.185] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:34.185] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.186] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:34.186] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0a15670a] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.186] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:34.186] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x0a15670a] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:34.186] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:34.186] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.188] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:34.188] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.202] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:34.203] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:34.203] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:34.203] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.204] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.204] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:34.204] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.204] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.205] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.206] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2f581b9f),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:34.206] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.206] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:34.206] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:34.207] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:34.210] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:34.211] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:34.381] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:34.382] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:34.382] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:34.382] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:34.382] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:34.382] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.383] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:34.383] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.383] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:34.383] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.384] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@19a40cfc),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:34.384] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.384] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:34.384] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:34.384] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:34.385] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:34.385] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:34.385] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.386] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:34.387] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6150818a] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.387] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6150818a] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:34.387] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:34.387] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:34.387] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:34.387] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:34.387] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.387] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:34.388] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:34.388] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.389] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.389] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:34.389] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1ebdff3b] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.390] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x1ebdff3b] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:34.390] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:34.390] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:34.390] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:34.390] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.400] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:34.400] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:34.400] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:34.401] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.401] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:34.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.401] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:34.403] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:34.403] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.403] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.404] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x56ad4264] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.404] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x56ad4264] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:34.404] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:34.405] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:34.405] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:34.482] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.482] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:34.482] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.483] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.483] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@45e4d960),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:34.483] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.483] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:34.483] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:34.484] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:34.484] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:34.484] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.484] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:34.484] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.485] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x337b5179] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.485] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:34.485] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x337b5179] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:34.485] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:34.485] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.485] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:34.485] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.500] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.500] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:34.501] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.502] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:34.502] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.502] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x17aedbde] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.502] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x17aedbde] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:34.502] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:34.502] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:34.502] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:34.503] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.503] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.503] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.503] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2013706e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:34.503] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.504] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:34.504] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:34.504] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:34.505] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:34.506] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:34.580] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:34.581] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:34.581] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:34.581] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:34.581] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:34.581] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.581] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:34.582] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:34.582] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.583] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:34.584] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4706e02e] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.584] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:34.584] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x4706e02e] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:34.584] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:34.584] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.598] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:34.599] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:34.599] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:34.599] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.599] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.600] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:34.600] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.600] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.601] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.601] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:34.601] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4e877a44),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:34.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.602] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:34.602] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:34.603] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [ERROR] [06/14/2012 08:19:34.603] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.603] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:34.603] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.604] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.604] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x206c2ea3] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.605] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x206c2ea3] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:34.605] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:34.605] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:34.605] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:34.608] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:34.608] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:34.777] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:34.778] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:34.778] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:34.778] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:34.778] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:34.778] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.778] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:34.779] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:34.779] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.780] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:34.780] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6411c21b] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.780] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6411c21b] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:34.780] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:34.780] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:34.780] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.796] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:34.796] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:34.796] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:34.796] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.796] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:34.796] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.796] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.797] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.797] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.798] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:34.798] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.798] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@34feb215),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:34.798] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.798] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:34.798] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:34.798] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [ERROR] [06/14/2012 08:19:34.799] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.799] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:34.799] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.799] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.799] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3d5d9be1] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.800] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:34.800] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3d5d9be1] already closed, dropping
[first] [INFO] [06/14/2012 08:19:34.800] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:34.800] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:34.801] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:34.801] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:34.877] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.877] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:34.877] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.877] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.877] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:34.878] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7ecdc97b),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:34.878] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.878] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:34.878] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.878] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.878] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:34.878] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.878] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:34.878] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7866eb46] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.878] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x7866eb46] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:34.879] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:34.879] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:34.879] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.879] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:34.879] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.896] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.896] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:34.897] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.897] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:34.897] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:34.897] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x78952527] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.898] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:34.898] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x78952527] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:34.898] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:34.898] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:34.898] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.898] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.898] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.899] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@1200d083),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:34.899] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.899] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:34.899] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:34.899] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:34.900] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:34.901] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:34.976] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:34.977] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:34.977] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:34.977] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:34.978] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:34.978] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:34.979] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:34.980] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x16fdf48d] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.980] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:34.980] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x16fdf48d] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:34.980] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:34.980] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:34.994] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:34.994] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:34.994] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:34.995] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.995] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:34.995] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:34.995] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:34.996] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:34.996] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:34.996] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7821f46a),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:34.997] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:34.997] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:34.997] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:34.997] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:34.997] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [DEBUG] [06/14/2012 08:19:34.998] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:34.998] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:35.001] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:35.001] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:35.009] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:35.010] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:35.010] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:35.192] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:35.192] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:35.192] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:35.192] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:35.192] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:35.193] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:35.193] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:35.193] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:35.193] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:35.194] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:35.194] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:35.195] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x778b3fee] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.195] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:35.195] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x778b3fee] already closed, dropping
[first] [INFO] [06/14/2012 08:19:35.195] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:35.195] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:35.207] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:35.207] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:35.207] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:35.207] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:35.207] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:35.207] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:35.207] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:35.208] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:35.208] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:35.209] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:35.209] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x138ba593] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.209] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:35.209] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x138ba593] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:35.209] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:35.210] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:35.291] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:35.291] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:35.292] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:35.292] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:35.293] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:35.293] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1afb7ac7] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.293] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:35.293] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x1afb7ac7] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:35.293] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:35.293] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:35.293] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:35.294] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:35.294] [ConvergenceSpec-akka.actor.default-dispatcher-8] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.294] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@62efae3b),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:35.294] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.295] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:35.295] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:35.295] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:35.296] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:35.296] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:35.305] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:35.306] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:35.306] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.306] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:35.306] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:35.306] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@ee67c12),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:35.306] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.307] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:35.307] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:35.307] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:35.307] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:35.307] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:35.307] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:35.307] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3ddb8962] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.308] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:35.308] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x3ddb8962] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:35.308] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:35.308] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:35.308] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:35.308] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:35.390] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:35.390] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:35.390] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:35.390] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:35.390] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:35.390] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:35.390] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:35.391] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:35.391] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.391] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4aa168c),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:35.391] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.392] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:35.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:35.392] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:35.394] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:35.394] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:35.405] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:35.406] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:35.406] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:35.406] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:35.406] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:35.406] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:35.406] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:35.407] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:35.407] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:35.408] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:35.409] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x754340bb] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x754340bb] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:35.409] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:35.409] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:35.409] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:35.589] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:35.589] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:35.589] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:35.589] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:35.589] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:35.589] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:35.590] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:35.590] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:35.591] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:35.591] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:35.592] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:35.593] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6815ee24] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.593] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6815ee24] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:35.593] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:35.593] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:35.593] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:35.593] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:35.593] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:35.593] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:35.594] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:35.594] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:35.595] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:35.596] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x64b6be69] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x64b6be69] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:35.596] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:35.596] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:35.596] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:35.603] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:35.603] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:35.603] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:35.603] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:35.603] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:35.603] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:35.603] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:35.604] [ConvergenceSpec-akka.actor.default-dispatcher-1] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:35.604] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:35.604] [ConvergenceSpec-akka.actor.default-dispatcher-1] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:35.605] [ConvergenceSpec-akka.actor.default-dispatcher-1] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1be0799a] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.605] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:35.605] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x1be0799a] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:35.605] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:35.605] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:35.687] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:35.688] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:35.688] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:35.689] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:35.689] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:35.689] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4e57dc21] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.689] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:35.689] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x4e57dc21] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:35.689] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:35.689] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:35.689] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:35.689] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:35.689] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.690] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@27077aa7),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:35.690] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.690] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:35.690] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:35.691] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:35.691] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:35.692] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:35.701] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:35.701] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:35.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:35.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:35.702] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5a20d10a),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:35.702] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.702] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:35.703] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:35.703] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:35.703] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:35.703] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:35.703] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:35.703] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x207c15c0] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.703] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x207c15c0] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:35.703] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:35.703] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:35.703] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:35.704] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:35.704] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:35.786] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:35.786] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:35.786] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:35.786] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:35.786] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:35.786] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:35.786] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:35.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:35.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:35.787] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:35.787] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@60f38b42),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:35.788] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.788] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:35.788] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:35.788] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:35.788] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:35.788] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:35.789] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:35.789] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:35.790] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x153bcbc8] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:35.790] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:35.790] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x153bcbc8] already closed, dropping
[first] [INFO] [06/14/2012 08:19:35.790] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:35.790] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:35.791] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:35.791] [ConvergenceSpec-akka.actor.default-dispatcher-1] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:35.801] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:35.801] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:35.801] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:35.801] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:35.801] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:35.801] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:35.801] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:35.802] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.802] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:35.802] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:35.802] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@76a40575),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:35.802] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.802] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:35.802] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:35.802] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [DEBUG] [06/14/2012 08:19:35.802] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:35.802] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:35.803] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:35.803] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:35.803] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [ERROR] [06/14/2012 08:19:35.803] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:35.803] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:35.803] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x0a62812d] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:35.804] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:35.804] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x0a62812d] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:35.804] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:35.804] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:35.805] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:35.805] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:35.998] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:35.999] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:35.999] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:35.999] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:35.999] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:35.999] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.000] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.000] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.000] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@5f2e712f),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.000] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.001] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:36.001] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.001] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:36.004] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:36.004] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.017] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:36.017] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:36.017] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:36.017] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.018] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.018] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:36.018] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:36.018] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:36.018] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:36.018] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.019] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:36.019] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x33d063fd] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.019] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x33d063fd] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:36.019] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:36.019] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:36.019] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:36.098] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.098] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.099] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.099] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.099] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:36.099] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6208f57a),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.099] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.099] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:36.099] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:36.099] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:36.099] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:36.099] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.100] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:36.100] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3a2c4ede] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.100] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:36.100] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x3a2c4ede] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:36.100] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:36.100] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:36.100] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:36.100] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.116] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:36.116] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:36.118] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.118] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:36.118] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.118] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x48899e6a] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.118] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x48899e6a] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:36.118] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:36.119] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:36.119] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:36.119] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:36.119] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:36.119] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.120] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@570f80a9),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.120] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.120] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:36.120] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.120] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:36.121] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:36.122] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:36.197] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:36.197] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:36.197] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:36.198] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:36.198] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:36.198] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.198] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.199] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.199] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:36.199] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.199] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@ecfeb11),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.199] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.200] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:36.200] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:36.200] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.200] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:36.201] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:36.201] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:36.202] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.202] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7451b0af] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.202] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x7451b0af] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:36.202] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:36.203] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:36.203] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:36.204] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:36.204] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.215] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:36.215] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:36.215] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:36.215] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.215] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:36.215] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.215] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:36.215] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:36.216] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:36.216] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.216] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:36.216] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7d9331eb] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.216] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:36.216] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x7d9331eb] already closed, dropping
[first] [INFO] [06/14/2012 08:19:36.216] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:36.216] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:36.394] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:36.394] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:36.395] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:36.395] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:36.395] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:36.395] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.395] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.396] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@3929df79),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.396] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.396] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:36.397] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.397] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:36.398] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:36.399] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.413] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:36.413] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:36.413] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:36.413] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:36.413] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.413] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.413] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:36.413] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:36.414] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.414] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:36.414] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:36.414] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2d205042),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.414] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:36.414] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.414] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:36.415] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:36.415] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.415] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [ERROR] [06/14/2012 08:19:36.415] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.415] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:36.415] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x24753433] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.415] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:36.416] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x24753433] already closed, dropping
[first] [INFO] [06/14/2012 08:19:36.416] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:36.416] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:36.418] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:36.418] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:36.494] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.494] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.495] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.495] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.495] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:36.495] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6c0e9e40),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.495] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [ERROR] [06/14/2012 08:19:36.495] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.495] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:36.495] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:36.495] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.495] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.495] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:36.496] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x33b7b32c] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.496] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:36.496] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x33b7b32c] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:36.496] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:36.496] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:36.496] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:36.496] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.512] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:36.512] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:36.513] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.513] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:36.513] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.513] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1f31cd12] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.513] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x1f31cd12] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:36.513] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:36.513] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:36.513] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:36.513] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:36.514] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:36.514] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.514] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@9f293df),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.514] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.514] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:36.514] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.514] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:36.515] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:36.515] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:36.593] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:36.593] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:36.593] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:36.593] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:36.593] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:36.593] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.594] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.594] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.594] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:36.594] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.594] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@28a2f6b),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.594] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.594] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:36.595] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:36.595] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.595] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:36.595] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:36.595] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:36.596] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.596] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3e2f1b1a] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.596] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3e2f1b1a] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:36.596] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:36.596] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:36.596] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:36.601] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:36.602] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.611] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:36.611] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:36.611] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:36.611] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.611] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.611] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:36.611] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:36.611] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:36.612] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:36.612] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.612] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:36.612] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x136c03ee] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.612] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:36.612] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x136c03ee] already closed, dropping
[first] [INFO] [06/14/2012 08:19:36.613] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:36.613] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:36.790] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:36.791] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:36.791] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:36.791] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:36.791] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:36.791] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.791] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:36.792] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:36.792] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:36.792] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x7f29b922] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[fourth] [DEBUG] [06/14/2012 08:19:36.793] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x7f29b922] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:36.793] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:36.793] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:36.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.793] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:36.793] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:36.794] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:36.794] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:36.794] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.795] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x21c8dfe6] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.795] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x21c8dfe6] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:36.795] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[fourth] [INFO] [06/14/2012 08:19:36.795] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[fourth] [DEBUG] [06/14/2012 08:19:36.795] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[first] [DEBUG] [06/14/2012 08:19:36.809] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:36.809] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:36.809] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:36.809] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.809] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:36.809] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.809] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:36.810] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:36.810] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.810] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2c8fc809),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.810] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.811] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:36.811] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.811] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:36.813] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:36.813] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:36.890] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.890] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.891] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.891] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.891] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@626287d3),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.891] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:36.891] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.892] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:36.892] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.892] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.892] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:36.892] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:36.892] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:36.892] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x32c3601b] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.892] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x32c3601b] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:36.892] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:36.893] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:36.893] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:36.893] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:36.893] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:36.908] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:36.908] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:36.909] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.909] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:36.909] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:36.909] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1395ddba] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.909] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x1395ddba] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:36.909] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:36.909] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:36.909] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:36.909] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:36.909] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:36.909] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.910] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@76a3c7b1),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.910] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.910] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:36.910] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.910] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:36.911] [ConvergenceSpec-6] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:36.911] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:36.988] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:36.989] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:36.989] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:36.989] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:36.989] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:36.989] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:36.989] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:36.989] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:36.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@21533b2c),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:36.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:36.990] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:36.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:36.990] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:36.992] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:36.992] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.007] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:37.007] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:37.007] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:37.007] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.008] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:37.008] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.008] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:37.008] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:37.008] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:37.008] [ConvergenceSpec-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.009] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:37.009] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x142b452b] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.009] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x142b452b] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:37.009] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:37.009] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:37.009] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:37.187] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:37.187] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:37.187] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:37.187] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:37.187] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:37.187] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:37.187] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:37.188] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:37.188] [ConvergenceSpec-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:37.188] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:37.188] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x10f0f6ac] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.188] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:37.188] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x10f0f6ac] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:37.188] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:37.188] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:37.205] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:37.206] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:37.206] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:37.206] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.206] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:37.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:37.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:37.206] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.207] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@a4aad7f),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:37.207] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.207] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:37.207] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:37.207] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [DEBUG] [06/14/2012 08:19:37.209] [ConvergenceSpec-7] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:37.209] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Receiving gossip from [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:37.286] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:37.286] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:37.286] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.286] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:37.286] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:37.287] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@a0ccc96),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:37.287] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.287] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:37.287] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.287] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:37.288] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:37.288] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:37.288] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:37.288] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4e4b9101] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.288] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:37.288] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x4e4b9101] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:37.288] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:37.288] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:37.289] [ConvergenceSpec-4] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:37.289] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.304] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:37.304] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:37.305] [ConvergenceSpec-akka.actor.default-dispatcher-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.305] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:37.305] [ConvergenceSpec-2] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.305] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2cc47220] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.305] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:37.305] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x2cc47220] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:37.305] [ConvergenceSpec-akka.actor.default-dispatcher-5] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:37.305] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:37.305] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:37.305] [ConvergenceSpec-akka.actor.default-dispatcher-5] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:37.306] [ConvergenceSpec-akka.actor.default-dispatcher-5] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.306] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@19509443),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:37.306] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.306] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:37.306] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:37.307] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:37.307] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:37.307] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:37.385] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:37.385] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:37.385] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:37.385] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:37.385] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:37.385] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:37.386] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:37.386] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.386] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:37.386] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:37.386] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@64b6be69),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:37.386] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.387] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:37.387] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[first] [DEBUG] [06/14/2012 08:19:37.387] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[first] [DEBUG] [06/14/2012 08:19:37.387] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:37.387] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3072, cap=3072)
[fourth] [ERROR] [06/14/2012 08:19:37.387] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:37.387] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.387] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:37.387] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=103, cap=103)
[fourth] [DEBUG] [06/14/2012 08:19:37.388] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:37.388] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x198f1327] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.388] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x198f1327] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:37.388] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:37.388] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:37.388] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:37.389] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:37.390] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.403] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:37.403] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:37.403] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:37.403] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.403] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:37.403] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.403] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:37.403] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:37.404] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:37.404] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.404] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:37.405] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x47f07360] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.405] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x47f07360] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:37.405] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:37.405] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:37.405] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:37.583] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:37.583] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:37.583] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:37.583] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:37.583] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:37.583] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:37.584] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:37.584] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.584] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:37.584] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:37.584] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@4e57dc21),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:37.584] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.584] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:37.584] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:37.585] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:37.585] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:37.585] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:37.585] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:37.585] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:37.586] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x6a3522b5] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.586] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x6a3522b5] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:37.586] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:37.586] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:37.586] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:37.587] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:37.587] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:37.601] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:37.601] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:37.601] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.601] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:37.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:37.601] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:37.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:37.602] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:37.602] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x2098746b] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.602] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x2098746b] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:37.603] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:37.603] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:37.603] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:37.682] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:37.682] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:37.682] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.682] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:37.682] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@6c908f05),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:37.682] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:37.682] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.682] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[first] [DEBUG] [06/14/2012 08:19:37.683] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:37.683] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.683] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:37.683] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:37.683] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:37.683] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x069dc8f2] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.683] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x069dc8f2] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:37.683] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:37.683] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:37.683] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:37.683] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:37.684] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.700] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:37.700] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:37.701] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.701] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [ERROR] [06/14/2012 08:19:37.701] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.701] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x49b3d1e5] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.701] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:37.701] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x49b3d1e5] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:37.701] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:37.701] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:37.701] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:37.701] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:37.701] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@2074714c),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:37.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.702] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:37.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:37.702] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:37.703] [ConvergenceSpec-2] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:37.703] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:37.781] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:37.781] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:37.781] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:37.781] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:37.781] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:37.781] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:37.781] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:37.782] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.782] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@41649a55),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:37.782] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.782] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:37.783] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:37.783] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:37.784] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:37.785] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.798] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:37.799] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:37.799] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:37.799] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.799] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.799] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:37.799] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:37.800] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:37.800] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:37.800] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.801] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:37.801] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x54ad00b2] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.801] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x54ad00b2] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:37.802] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:37.802] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:37.802] [ConvergenceSpec-akka.actor.default-dispatcher-8] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:37.979] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:37.980] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:37.980] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:37.980] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:37.980] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:37.980] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:37.981] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:37.981] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:37.982] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@7f39ebdb),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:37.982] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.982] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:37.982] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:37.983] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:37.985] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:37.986] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.997] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:37.997] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:37.997] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:37.997] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.997] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:37.997] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:37.997] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:37.997] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:37.998] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:37.998] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:37.998] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:37.999] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x31d47845] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:37.999] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@third]
[first] [DEBUG] [06/14/2012 08:19:37.999] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x31d47845] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:37.999] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [INFO] [06/14/2012 08:19:37.999] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:37.999] [ConvergenceSpec-akka.actor.default-dispatcher-4] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@third/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:37.999] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:37.999] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@third|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:37.999] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:37.999] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@third: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:38.000] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@third/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:38.000] [ConvergenceSpec-akka.actor.default-dispatcher-4] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x33e2ad75] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:38.000] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x33e2ad75] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:38.000] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@third]
[first] [INFO] [06/14/2012 08:19:38.000] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@third
[first] [DEBUG] [06/14/2012 08:19:38.000] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@third] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:38.078] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:38.078] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:38.079] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.079] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:38.079] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:38.079] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@33abb81e),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:38.080] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:38.080] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [ERROR] [06/14/2012 08:19:38.080] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:38.080] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:38.081] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:38.081] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:38.081] [ConvergenceSpec-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:38.081] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4ec4d412] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.081] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x4ec4d412] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:38.081] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:38.081] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:38.082] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:38.082] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@fourth) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:38.082] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:38.096] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:38.096] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:38.097] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:38.097] [ConvergenceSpec-4] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:38.097] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:38.097] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x78e782a8] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:38.098] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:38.098] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x78e782a8] already closed, dropping
[first] [DEBUG] [06/14/2012 08:19:38.098] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Heartbeat to [Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]]
[first] [INFO] [06/14/2012 08:19:38.098] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:38.098] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:38.098] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@first) from None to Actor[akka://ConvergenceSpec@fourth/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:38.098] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[first] [DEBUG] [06/14/2012 08:19:38.098] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@468864af),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[first] [DEBUG] [06/14/2012 08:19:38.098] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-snd3] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.099] [New I/O client worker #1-1] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:38.099] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@6fef3212),BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:38.099] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-rcv1] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=468, cap=468)
[fourth] [DEBUG] [06/14/2012 08:19:38.100] [ConvergenceSpec-5] [NettyRemoteTransport(akka://ConvergenceSpec@fourth)] received local message RemoteMessage: Heartbeat(akka://ConvergenceSpec@first) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@fourth/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[fourth] [DEBUG] [06/14/2012 08:19:38.100] [ConvergenceSpec-akka.actor.default-dispatcher-4] [akka.cluster.FailureDetectorPuppet] Heart beat from cluster node[akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:38.177] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:38.177] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:38.177] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:38.177] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:38.177] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:38.178] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:38.178] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:38.178] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.179] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:38.179] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:38.179] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@8523ca2),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:38.179] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.179] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [DEBUG] [06/14/2012 08:19:38.180] [New I/O server worker #1-3] [FailureInjector(akka://ConvergenceSpec)] upstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] RECEIVED: BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[first] [DEBUG] [06/14/2012 08:19:38.180] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@40b890dc,Receive,Some(org.jboss.netty.channel.SucceededChannelFuture@152e7cb6),BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [ERROR] [06/14/2012 08:19:38.180] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:38.180] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/127.0.0.1:61433-rcv3] sending msg (PassThrough): BigEndianHeapChannelBuffer(ridx=0, widx=3175, cap=3175)
[fourth] [ERROR] [06/14/2012 08:19:38.180] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:38.181] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:38.182] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x4f0ab3f2] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.182] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x4f0ab3f2] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:38.182] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:38.182] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:38.182] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:38.183] [ConvergenceSpec-3] [NettyRemoteTransport(akka://ConvergenceSpec@first)] received local message RemoteMessage: GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) to Actor[akka://ConvergenceSpec/system/cluster/gossip]<+{akka://ConvergenceSpec@first/system/cluster/gossip} from Actor[akka://ConvergenceSpec/deadLetters]
[first] [DEBUG] [06/14/2012 08:19:38.184] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Receiving gossip from [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:38.195] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Initiating new round of gossip
[first] [DEBUG] [06/14/2012 08:19:38.195] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[first] [DEBUG] [06/14/2012 08:19:38.195] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:38.195] [ConvergenceSpec-akka.actor.default-dispatcher-8] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - No cluster convergence, due to unreachable nodes [Set(Member(address = akka://ConvergenceSpec@third, status = Up))].
[first] [DEBUG] [06/14/2012 08:19:38.195] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:38.196] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[first] [DEBUG] [06/14/2012 08:19:38.196] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[first] [DEBUG] [06/14/2012 08:19:38.196] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[first] [ERROR] [06/14/2012 08:19:38.196] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [ERROR] [06/14/2012 08:19:38.196] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[first] [DEBUG] [06/14/2012 08:19:38.197] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@first,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[first] [DEBUG] [06/14/2012 08:19:38.197] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x3ba76e19] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[first] [DEBUG] [06/14/2012 08:19:38.197] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[first] [DEBUG] [06/14/2012 08:19:38.197] [ConvergenceSpec-akka.actor.default-dispatcher-8] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x3ba76e19] already closed, dropping
[first] [INFO] [06/14/2012 08:19:38.197] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[first] [DEBUG] [06/14/2012 08:19:38.197] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[first] [DEBUG] [06/14/2012 08:19:38.377] [ConvergenceSpec-akka.actor.default-dispatcher-4] [PlayerHandler(akka://ConvergenceSpec)] disconnected from /127.0.0.1:4711
[first] [DEBUG] [06/14/2012 08:19:38.377] [ConvergenceSpec-akka.actor.default-dispatcher-8] [ConductorHandler(akka://ConvergenceSpec)] disconnect from /127.0.0.1:61423
[fourth] [DEBUG] [06/14/2012 08:19:38.377] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] disconnected from localhost/127.0.0.1:4711
[first] [DEBUG] [06/14/2012 08:19:38.377] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ConductorHandler(akka://ConvergenceSpec)] disconnect from /127.0.0.1:61429
[fourth] [DEBUG] [06/14/2012 08:19:38.379] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 :> localhost/127.0.0.1:4711] unbound
[fourth] [DEBUG] [06/14/2012 08:19:38.379] [New I/O client worker #2-1] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x05c09624, /127.0.0.1:61429 :> localhost/127.0.0.1:4711] closed
[first] [WARN] [06/14/2012 08:19:38.380] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61429-server4] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: ClientDisconnected
[first] [WARN] [06/14/2012 08:19:38.380] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/controller/127.0.0.1:61423-server1] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: ClientDisconnected
[first] [DEBUG] [06/14/2012 08:19:38.380] [ConvergenceSpec-akka.actor.default-dispatcher-4] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 :> /127.0.0.1:4711] unbound
[first] [DEBUG] [06/14/2012 08:19:38.380] [ConvergenceSpec-akka.actor.default-dispatcher-4] [PlayerHandler(akka://ConvergenceSpec)] channel [id: 0x0a1e940b, /127.0.0.1:61423 :> /127.0.0.1:4711] closed
[first] [WARN] [06/14/2012 08:19:38.380] [ConvergenceSpec-akka.actor.default-dispatcher-5] [akka://ConvergenceSpec/user/TestConductorClient] received dead letter from Actor[akka://ConvergenceSpec/deadLetters]: PoisonPill
[first] [DEBUG] [06/14/2012 08:19:38.382] [ConvergenceSpec-akka.actor.default-dispatcher-8] [EventStream] shutting down: StandardOutLogger started
[first] [DEBUG] [06/14/2012 08:19:38.382] [ConvergenceSpec-akka.actor.default-dispatcher-8] [EventStream] shutting down: StandardOutLogger started
[first] [DEBUG] [06/14/2012 08:19:38.383] [ConvergenceSpec-akka.actor.default-dispatcher-8] [EventStream] all default loggers stopped
[first] [INFO] [06/14/2012 08:19:38.386] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@first] - Shutting down cluster Node and cluster daemons...
[first] - not move a new joining node to Up while there is no convergence *** FAILED ***
[first]   java.lang.AssertionError: assertion failed: timeout 19799426 microseconds expired
[first]   at scala.Predef$.assert(Predef.scala:160)
[first]   at akka.testkit.TestKitBase$class.poll$1(TestKit.scala:185)
[first]   at akka.testkit.TestKitBase$class.awaitCond(TestKit.scala:191)
[first]   at akka.testkit.TestKit.awaitCond(TestKit.scala:600)
[first]   at akka.cluster.MultiNodeClusterSpec$class.awaitSeenSameState(MultiNodeClusterSpec.scala:133)
[first]   at akka.cluster.ConvergenceSpec.awaitSeenSameState(ConvergenceSpec.scala:34)
[first]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$assertNotMovedUp$1$1.apply$mcV$sp(ConvergenceSpec.scala:100)
[first]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$assertNotMovedUp$1$1.apply(ConvergenceSpec.scala:98)
[first]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$assertNotMovedUp$1$1.apply(ConvergenceSpec.scala:98)
[first]   at akka.testkit.TestKitBase$class.within(TestKit.scala:222)
[first]   at akka.testkit.TestKit.within(TestKit.scala:600)
[first]   at akka.testkit.TestKitBase$class.within(TestKit.scala:236)
[first]   at akka.testkit.TestKit.within(TestKit.scala:600)
[first]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.assertNotMovedUp$1(ConvergenceSpec.scala:98)
[first]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$apply$mcV$sp$13$$anonfun$apply$mcV$sp$14.apply$mcVI$sp(ConvergenceSpec.scala:113)
[first]   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
[first]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$apply$mcV$sp$13.apply$mcV$sp(ConvergenceSpec.scala:111)
[first]   at akka.remote.testkit.MultiNodeSpec.runOn(MultiNodeSpec.scala:177)
[first]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.apply$mcV$sp(ConvergenceSpec.scala:110)
[first]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.apply(ConvergenceSpec.scala:84)
[first]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.apply(ConvergenceSpec.scala:84)
[first]   at org.scalatest.WordSpec$$anon$2.apply(WordSpec.scala:2161)
[first]   at org.scalatest.Suite$class.withFixture(Suite.scala:1968)
[first]   at akka.testkit.AkkaSpec.withFixture(AkkaSpec.scala:59)
[first]   at org.scalatest.WordSpec$class.invokeWithFixture$1(WordSpec.scala:2158)
[first]   at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[first]   at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[first]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:168)
[first]   at org.scalatest.WordSpec$class.runTest(WordSpec.scala:2167)
[first]   at akka.testkit.AkkaSpec.runTest(AkkaSpec.scala:59)
[first]   at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[first]   at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[first]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:226)
[first]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[first]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[first]   at scala.collection.immutable.List.foreach(List.scala:76)
[first]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[first]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:231)
[first]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[first]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[first]   at scala.collection.immutable.List.foreach(List.scala:76)
[first]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[first]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:282)
[first]   at org.scalatest.WordSpec$class.runTests(WordSpec.scala:2232)
[first]   at akka.testkit.AkkaSpec.runTests(AkkaSpec.scala:59)
[first]   at org.scalatest.Suite$class.run(Suite.scala:2286)
[first]   at akka.testkit.AkkaSpec.org$scalatest$WordSpec$$super$run(AkkaSpec.scala:59)
[first]   at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[first]   at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[first]   at org.scalatest.SuperEngine.runImpl(Engine.scala:318)
[first]   at org.scalatest.WordSpec$class.run(WordSpec.scala:2279)
[first]   at akka.testkit.AkkaSpec.org$scalatest$BeforeAndAfterAll$$super$run(AkkaSpec.scala:59)
[first]   at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:213)
[first]   at akka.testkit.AkkaSpec.run(AkkaSpec.scala:59)
[first]   at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:59)
[first]   at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1517)
[first]   at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1514)
[first]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[first]   at scala.collection.immutable.List.foreach(List.scala:76)
[first]   at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1514)
[first]   at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:584)
[first]   at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:583)
[first]   at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1558)
[first]   at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:582)
[first]   at org.scalatest.tools.Runner$.main(Runner.scala:466)
[first]   at org.scalatest.tools.Runner.main(Runner.scala)
[first]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[first]   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[first]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[first]   at java.lang.reflect.Method.invoke(Method.java:597)
[first]   at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
[first]   at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
[first]   at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
[first]   at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
[first]   at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
[first]   at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
[first]   at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
[first]   at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
[first]   at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
[first]   at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
[first]   at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[first] [DEBUG] [06/14/2012 08:19:38.390] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [PassiveRemoteClient@akka://ConvergenceSpec@fourth]
[first] [INFO] [06/14/2012 08:19:38.390] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@fourth
[first] [DEBUG] [06/14/2012 08:19:38.390] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [PassiveRemoteClient@akka://ConvergenceSpec@fourth] has been shut down
[first] [DEBUG] [06/14/2012 08:19:38.391] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x43277a30, localhost/127.0.0.1:61433 => /127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=42, cap=42, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.408] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Initiating new round of gossip
[fourth] [DEBUG] [06/14/2012 08:19:38.408] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@first]
[fourth] [DEBUG] [06/14/2012 08:19:38.408] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:38.408] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka.cluster.FailureDetectorPuppet] isAvailable: Cluster node IS NOT available [akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:38.408] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@first, akka://ConvergenceSpec@second, akka://ConvergenceSpec@fourth]
[fourth] [DEBUG] [06/14/2012 08:19:38.408] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:38.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:38.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Selecting random node to gossip to [akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:38.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Gossiping to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:38.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@51493995),CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:38.409] [ConvergenceSpec-akka.actor.default-dispatcher-7] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.409] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [ERROR] [06/14/2012 08:19:38.410] [ConvergenceSpec-akka.actor.default-dispatcher-3] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [ERROR] [06/14/2012 08:19:38.410] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:38.410] [ConvergenceSpec-akka.actor.default-dispatcher-3] [RemoteClient(akka://ConvergenceSpec)] Sending message GossipEnvelope(akka://ConvergenceSpec@fourth,Gossip(overview = GossipOverview(seen = [akka://ConvergenceSpec@fourth -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919), akka://ConvergenceSpec@first -> VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919)], unreachable = [Member(address = akka://ConvergenceSpec@third, status = Up)]), members = [Member(address = akka://ConvergenceSpec@first, status = Up), Member(address = akka://ConvergenceSpec@second, status = Up), Member(address = akka://ConvergenceSpec@fourth, status = Up)], meta = [], version = VectorClock(Node(266d4125d05511b21390a70cf021ffc5) -> 00000137e9a56098, Node(84eaec785751a55240075d71f7e29938) -> 00000137e9a5588c, Node(78b82d21ce185801aa210dc681a06b00) -> 00000137e9a5623b, Node(c747d1bd83d00e993fcd418f2e67a10f) -> 00000137e9a55919))) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:38.410] [ConvergenceSpec-akka.actor.default-dispatcher-3] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x1e638ee4] WRITE: CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.410] [ConvergenceSpec-akka.actor.default-dispatcher-6] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=3175, cap=3175, components=2) in direction Send when socket [id: 0x1e638ee4] already closed, dropping
[fourth] [DEBUG] [06/14/2012 08:19:38.410] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [INFO] [06/14/2012 08:19:38.411] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:38.411] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:38.507] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@first/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:38.507] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@first/system/cluster/gossip]
[fourth] [DEBUG] [06/14/2012 08:19:38.507] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x45d1c3cd, /127.0.0.1:61433 => localhost/127.0.0.1:61416] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.507] [ConvergenceSpec-akka.actor.default-dispatcher-7] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Heartbeat to [Actor[akka://ConvergenceSpec@second/system/cluster/gossip]]
[fourth] [DEBUG] [06/14/2012 08:19:38.508] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] processing Event(Send(org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3529c445,Send,Some(org.jboss.netty.channel.DefaultChannelFuture@557531fd),CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)),Data(0,-1.0,Queue())) from Actor[akka://ConvergenceSpec/system/FailureInjector]
[fourth] [DEBUG] [06/14/2012 08:19:38.508] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Starting remote client connection to [akka://ConvergenceSpec@second|localhost/127.0.0.1]
[fourth] [DEBUG] [06/14/2012 08:19:38.508] [ConvergenceSpec-akka.actor.default-dispatcher-3] [akka://ConvergenceSpec/system/FailureInjector/first-snd1] sending msg (PassThrough): CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [ERROR] [06/14/2012 08:19:38.508] [ConvergenceSpec-akka.actor.default-dispatcher-7] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:38.509] [ConvergenceSpec-akka.actor.default-dispatcher-7] [RemoteClient(akka://ConvergenceSpec)] Sending message Heartbeat(akka://ConvergenceSpec@fourth) from None to Actor[akka://ConvergenceSpec@second/system/cluster/gossip]
[fourth] [ERROR] [06/14/2012 08:19:38.509] [ConvergenceSpec-5] [ActorSystem(ConvergenceSpec)] RemoteClientError@akka://ConvergenceSpec@second: Error[java.net.ConnectException: Connection refused]
[fourth] [DEBUG] [06/14/2012 08:19:38.509] [ConvergenceSpec-akka.actor.default-dispatcher-7] [FailureInjector(akka://ConvergenceSpec)] downstream(queued): [id: 0x012504e0] WRITE: CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2)
[fourth] [DEBUG] [06/14/2012 08:19:38.509] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@second]
[fourth] [DEBUG] [06/14/2012 08:19:38.509] [ConvergenceSpec-akka.actor.default-dispatcher-1] [akka://ConvergenceSpec/system/FailureInjector] sending CompositeChannelBuffer(ridx=0, widx=468, cap=468, components=2) in direction Send when socket [id: 0x012504e0] already closed, dropping
[fourth] [INFO] [06/14/2012 08:19:38.509] [ConvergenceSpec-akka.actor.default-dispatcher-4] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@second
[fourth] [DEBUG] [06/14/2012 08:19:38.509] [ConvergenceSpec-akka.actor.default-dispatcher-4] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@second] has been shut down
[fourth] [DEBUG] [06/14/2012 08:19:38.565] [ConvergenceSpec-akka.actor.default-dispatcher-4] [EventStream] shutting down: StandardOutLogger started
[fourth] [DEBUG] [06/14/2012 08:19:38.565] [ConvergenceSpec-akka.actor.default-dispatcher-4] [EventStream] shutting down: StandardOutLogger started
[fourth] [DEBUG] [06/14/2012 08:19:38.567] [ConvergenceSpec-akka.actor.default-dispatcher-4] [EventStream] all default loggers stopped
[fourth] - not move a new joining node to Up while there is no convergence *** FAILED ***
[fourth]   java.lang.AssertionError: assertion failed: timeout 19497114 microseconds expired
[fourth]   at scala.Predef$.assert(Predef.scala:160)
[fourth]   at akka.testkit.TestKitBase$class.poll$1(TestKit.scala:185)
[fourth]   at akka.testkit.TestKitBase$class.awaitCond(TestKit.scala:191)
[fourth]   at akka.testkit.TestKit.awaitCond(TestKit.scala:600)
[fourth]   at akka.cluster.MultiNodeClusterSpec$class.awaitSeenSameState(MultiNodeClusterSpec.scala:133)
[fourth]   at akka.cluster.ConvergenceSpec.awaitSeenSameState(ConvergenceSpec.scala:34)
[fourth]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$assertNotMovedUp$1$1.apply$mcV$sp(ConvergenceSpec.scala:100)
[fourth]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$assertNotMovedUp$1$1.apply(ConvergenceSpec.scala:98)
[fourth]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$assertNotMovedUp$1$1.apply(ConvergenceSpec.scala:98)
[fourth]   at akka.testkit.TestKitBase$class.within(TestKit.scala:222)
[fourth]   at akka.testkit.TestKit.within(TestKit.scala:600)
[fourth]   at akka.testkit.TestKitBase$class.within(TestKit.scala:236)
[fourth]   at akka.testkit.TestKit.within(TestKit.scala:600)
[fourth]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.assertNotMovedUp$1(ConvergenceSpec.scala:98)
[fourth]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$apply$mcV$sp$13$$anonfun$apply$mcV$sp$14.apply$mcVI$sp(ConvergenceSpec.scala:113)
[fourth]   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:78)
[fourth]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10$$anonfun$apply$mcV$sp$13.apply$mcV$sp(ConvergenceSpec.scala:111)
[fourth]   at akka.remote.testkit.MultiNodeSpec.runOn(MultiNodeSpec.scala:177)
[fourth]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.apply$mcV$sp(ConvergenceSpec.scala:110)
[fourth]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.apply(ConvergenceSpec.scala:84)
[fourth]   at akka.cluster.ConvergenceSpec$$anonfun$1$$anonfun$apply$mcV$sp$10.apply(ConvergenceSpec.scala:84)
[fourth]   at org.scalatest.WordSpec$$anon$2.apply(WordSpec.scala:2161)
[fourth]   at org.scalatest.Suite$class.withFixture(Suite.scala:1968)
[fourth]   at akka.testkit.AkkaSpec.withFixture(AkkaSpec.scala:59)
[fourth]   at org.scalatest.WordSpec$class.invokeWithFixture$1(WordSpec.scala:2158)
[fourth]   at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[fourth]   at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[fourth]   at org.scalatest.SuperEngine.runTestImpl(Engine.scala:168)
[fourth]   at org.scalatest.WordSpec$class.runTest(WordSpec.scala:2167)
[fourth]   at akka.testkit.AkkaSpec.runTest(AkkaSpec.scala:59)
[fourth]   at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[fourth]   at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[fourth]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:226)
[fourth]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[fourth]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[fourth]   at scala.collection.immutable.List.foreach(List.scala:76)
[fourth]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[fourth]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:231)
[fourth]   at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[fourth]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[fourth]   at scala.collection.immutable.List.foreach(List.scala:76)
[fourth]   at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[fourth]   at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:282)
[fourth]   at org.scalatest.WordSpec$class.runTests(WordSpec.scala:2232)
[fourth]   at akka.testkit.AkkaSpec.runTests(AkkaSpec.scala:59)
[fourth]   at org.scalatest.Suite$class.run(Suite.scala:2286)
[fourth]   at akka.testkit.AkkaSpec.org$scalatest$WordSpec$$super$run(AkkaSpec.scala:59)
[fourth]   at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[fourth]   at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[fourth]   at org.scalatest.SuperEngine.runImpl(Engine.scala:318)
[fourth]   at org.scalatest.WordSpec$class.run(WordSpec.scala:2279)
[fourth]   at akka.testkit.AkkaSpec.org$scalatest$BeforeAndAfterAll$$super$run(AkkaSpec.scala:59)
[fourth]   at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:213)
[fourth]   at akka.testkit.AkkaSpec.run(AkkaSpec.scala:59)
[fourth]   at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:59)
[fourth]   at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1517)
[fourth]   at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1514)
[fourth]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[fourth]   at scala.collection.immutable.List.foreach(List.scala:76)
[fourth]   at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1514)
[fourth]   at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:584)
[fourth]   at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:583)
[fourth]   at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1558)
[fourth]   at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:582)
[fourth]   at org.scalatest.tools.Runner$.main(Runner.scala:466)
[fourth]   at org.scalatest.tools.Runner.main(Runner.scala)
[fourth]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[fourth]   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[fourth]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[fourth]   at java.lang.reflect.Method.invoke(Method.java:597)
[fourth]   at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
[fourth]   at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
[fourth]   at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
[fourth]   at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
[fourth]   at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
[fourth]   at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
[fourth]   at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
[fourth]   at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
[fourth]   at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
[fourth]   at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
[fourth]   at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[fourth] [INFO] [06/14/2012 08:19:38.570] [ConvergenceSpec-akka.actor.default-dispatcher-6] [Node(akka://ConvergenceSpec)] Cluster Node [akka://ConvergenceSpec@fourth] - Shutting down cluster Node and cluster daemons...
[fourth] [DEBUG] [06/14/2012 08:19:38.577] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] Shutting down remote client [ActiveRemoteClient@akka://ConvergenceSpec@first]
[fourth] [INFO] [06/14/2012 08:19:38.577] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteClientShutdown@akka://ConvergenceSpec@first
[fourth] [DEBUG] [06/14/2012 08:19:38.577] [ConvergenceSpec-akka.actor.default-dispatcher-6] [FailureInjector(akka://ConvergenceSpec)] disconnected from [id: 0x45d1c3cd, /127.0.0.1:61433 :> localhost/127.0.0.1:61416]
[fourth] [DEBUG] [06/14/2012 08:19:38.577] [ConvergenceSpec-akka.actor.default-dispatcher-6] [RemoteClient(akka://ConvergenceSpec)] [ActiveRemoteClient@akka://ConvergenceSpec@first] has been shut down
[fourth] [INFO] [06/14/2012 08:19:38.580] [ConvergenceSpec-akka.actor.default-dispatcher-6] [ActorSystem(ConvergenceSpec)] RemoteServerShutdown@akka://ConvergenceSpec@fourth
[error] Failed: akka.cluster.ConvergenceWithFailureDetectorPuppetMultiJvmNode1
[error] Failed: akka.cluster.ConvergenceWithFailureDetectorPuppetMultiJvmNode2
[error] Failed: akka.cluster.ConvergenceWithFailureDetectorPuppetMultiJvmNode4
[success] Total time: 25 s, completed Jun 14, 2012 8:19:38 AM
akka-cluster >