[info] * akka.cluster.LeaderElectionWithFailureDetectorPuppet
[JVM-Node1] [INFO] [10/02/2012 04:13:35.705] [main] [LeaderElectionWithFailureDetectorPuppetMultiJvmNode1(akka://LeaderElectionSpec)] Role [controller] started with address [akka://LeaderElectionSpec@localhost:42898]
[JVM-Node1] LeaderElectionWithFailureDetectorPuppetMultiJvmNode1:
[JVM-Node2] [INFO] [10/02/2012 04:13:35.786] [main] [LeaderElectionWithFailureDetectorPuppetMultiJvmNode2(akka://LeaderElectionSpec)] Role [first] started with address [akka://LeaderElectionSpec@localhost:59476]
[JVM-Node4] [INFO] [10/02/2012 04:13:35.786] [main] [LeaderElectionWithFailureDetectorPuppetMultiJvmNode4(akka://LeaderElectionSpec)] Role [third] started with address [akka://LeaderElectionSpec@localhost:46933]
[JVM-Node1] A cluster of four nodes 
[JVM-Node5] [INFO] [10/02/2012 04:13:35.800] [main] [LeaderElectionWithFailureDetectorPuppetMultiJvmNode5(akka://LeaderElectionSpec)] Role [fourth] started with address [akka://LeaderElectionSpec@localhost:41943]
[JVM-Node3] [INFO] [10/02/2012 04:13:35.806] [main] [LeaderElectionWithFailureDetectorPuppetMultiJvmNode3(akka://LeaderElectionSpec)] Role [second] started with address [akka://LeaderElectionSpec@localhost:36354]
[JVM-Node4] LeaderElectionWithFailureDetectorPuppetMultiJvmNode4:
[JVM-Node2] LeaderElectionWithFailureDetectorPuppetMultiJvmNode2:
[JVM-Node4] A cluster of four nodes 
[JVM-Node5] LeaderElectionWithFailureDetectorPuppetMultiJvmNode5:
[JVM-Node3] LeaderElectionWithFailureDetectorPuppetMultiJvmNode3:
[JVM-Node2] A cluster of four nodes 
[JVM-Node3] A cluster of four nodes 
[JVM-Node5] A cluster of four nodes 
[JVM-Node2] [INFO] [10/02/2012 04:13:35.917] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:59476] - is starting up...
[JVM-Node2] [INFO] [10/02/2012 04:13:35.919] [main] [Cluster(akka://LeaderElectionSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[JVM-Node2] [INFO] [10/02/2012 04:13:35.973] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:59476] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node2] [INFO] [10/02/2012 04:13:35.973] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:59476] - has started up successfully
[JVM-Node2] [INFO] [10/02/2012 04:13:35.986] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [akka://LeaderElectionSpec/system/cluster/metrics] Hyperic SIGAR was not found on the classpath or not installed properly. Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriateplatform-specific native libary to 'java.library.path'.
[JVM-Node2] [INFO] [10/02/2012 04:13:35.997] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [akka://LeaderElectionSpec/system/cluster/metrics] Metrics collection has started successfully on node [akka://LeaderElectionSpec@localhost:59476]
[JVM-Node2] [WARN] [10/02/2012 04:13:36.027] [LeaderElectionSpec-akka.actor.default-dispatcher-2] [akka://LeaderElectionSpec/system/cluster/core] unhandled message from Actor[akka://LeaderElectionSpec/system/cluster/core]: JoinSeedNodes(Vector())
[JVM-Node5] [INFO] [10/02/2012 04:13:36.115] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:41943] - is starting up...
[JVM-Node3] [INFO] [10/02/2012 04:13:36.115] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:36354] - is starting up...
[JVM-Node4] [INFO] [10/02/2012 04:13:36.115] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:46933] - is starting up...
[JVM-Node3] [INFO] [10/02/2012 04:13:36.117] [main] [Cluster(akka://LeaderElectionSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[JVM-Node5] [INFO] [10/02/2012 04:13:36.117] [main] [Cluster(akka://LeaderElectionSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[JVM-Node4] [INFO] [10/02/2012 04:13:36.117] [main] [Cluster(akka://LeaderElectionSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <=  'akka.cluster.scheduler.tick-duration' [33 ms].
[JVM-Node5] [INFO] [10/02/2012 04:13:36.169] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:41943] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node5] [INFO] [10/02/2012 04:13:36.169] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:41943] - has started up successfully
[JVM-Node4] [INFO] [10/02/2012 04:13:36.171] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:46933] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node4] [INFO] [10/02/2012 04:13:36.171] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:46933] - has started up successfully
[JVM-Node3] [INFO] [10/02/2012 04:13:36.173] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:36354] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node3] [INFO] [10/02/2012 04:13:36.173] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@localhost:36354] - has started up successfully
[JVM-Node5] [INFO] [10/02/2012 04:13:36.178] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [akka://LeaderElectionSpec/system/cluster/metrics] Hyperic SIGAR was not found on the classpath or not installed properly. Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriateplatform-specific native libary to 'java.library.path'.
[JVM-Node4] [INFO] [10/02/2012 04:13:36.181] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [akka://LeaderElectionSpec/system/cluster/metrics] Hyperic SIGAR was not found on the classpath or not installed properly. Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriateplatform-specific native libary to 'java.library.path'.
[JVM-Node3] [INFO] [10/02/2012 04:13:36.182] [LeaderElectionSpec-akka.actor.default-dispatcher-3] [akka://LeaderElectionSpec/system/cluster/metrics] Hyperic SIGAR was not found on the classpath or not installed properly. Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriateplatform-specific native libary to 'java.library.path'.
[JVM-Node4] [INFO] [10/02/2012 04:13:36.186] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [akka://LeaderElectionSpec/system/cluster/metrics] Metrics collection has started successfully on node [akka://LeaderElectionSpec@localhost:46933]
[JVM-Node5] [INFO] [10/02/2012 04:13:36.189] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [akka://LeaderElectionSpec/system/cluster/metrics] Metrics collection has started successfully on node [akka://LeaderElectionSpec@localhost:41943]
[JVM-Node3] [INFO] [10/02/2012 04:13:36.191] [LeaderElectionSpec-akka.actor.default-dispatcher-3] [akka://LeaderElectionSpec/system/cluster/metrics] Metrics collection has started successfully on node [akka://LeaderElectionSpec@localhost:36354]
[JVM-Node5] [WARN] [10/02/2012 04:13:36.215] [LeaderElectionSpec-akka.actor.default-dispatcher-3] [akka://LeaderElectionSpec/system/cluster/core] unhandled message from Actor[akka://LeaderElectionSpec/system/cluster/core]: JoinSeedNodes(Vector())
[JVM-Node4] [WARN] [10/02/2012 04:13:36.217] [LeaderElectionSpec-akka.actor.default-dispatcher-3] [akka://LeaderElectionSpec/system/cluster/core] unhandled message from Actor[akka://LeaderElectionSpec/system/cluster/core]: JoinSeedNodes(Vector())
[JVM-Node3] [WARN] [10/02/2012 04:13:36.218] [LeaderElectionSpec-akka.actor.default-dispatcher-5] [akka://LeaderElectionSpec/system/cluster/core] unhandled message from Actor[akka://LeaderElectionSpec/system/cluster/core]: JoinSeedNodes(Vector())
[JVM-Node2] [INFO] [10/02/2012 04:13:36.332] [LeaderElectionSpec-akka.actor.default-dispatcher-5] [akka://LeaderElectionSpec/system/cluster/core] Cluster Node [akka://LeaderElectionSpec@localhost:59476] - Leader is moving node [akka://LeaderElectionSpec@localhost:59476] from JOINING to UP
[JVM-Node4] [WARN] [10/02/2012 04:13:36.535] [LeaderElectionSpec-akka.actor.default-dispatcher-3] [akka://LeaderElectionSpec/system/FailureInjector] installing context org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@42143753 instead of org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@3d3c4c09 for address akka://LeaderElectionSpec@localhost:41943
[JVM-Node5] [WARN] [10/02/2012 04:13:36.562] [LeaderElectionSpec-akka.actor.default-dispatcher-1] [akka://LeaderElectionSpec/system/FailureInjector] installing context org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@1fb88122 instead of org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@a25ef73 for address akka://LeaderElectionSpec@localhost:46933
[JVM-Node3] [INFO] [10/02/2012 04:13:36.920] [LeaderElectionSpec-akka.actor.default-dispatcher-1] [akka://LeaderElectionSpec/system/cluster/core] Cluster Node [akka://LeaderElectionSpec@localhost:36354] - Leader is moving node [akka://LeaderElectionSpec@localhost:36354] from JOINING to UP
[JVM-Node3] [INFO] [10/02/2012 04:13:36.920] [LeaderElectionSpec-akka.actor.default-dispatcher-1] [akka://LeaderElectionSpec/system/cluster/core] Cluster Node [akka://LeaderElectionSpec@localhost:36354] - Leader is moving node [akka://LeaderElectionSpec@localhost:41943] from JOINING to UP
[JVM-Node3] [INFO] [10/02/2012 04:13:36.921] [LeaderElectionSpec-akka.actor.default-dispatcher-1] [akka://LeaderElectionSpec/system/cluster/core] Cluster Node [akka://LeaderElectionSpec@localhost:36354] - Leader is moving node [akka://LeaderElectionSpec@localhost:46933] from JOINING to UP
[JVM-Node3] [WARN] [10/02/2012 04:13:36.925] [LeaderElectionSpec-akka.actor.default-dispatcher-6] [akka://LeaderElectionSpec@localhost:41943/system/cluster/core] received dead letter from Actor[akka://LeaderElectionSpec/system/cluster/core/coreSender]: GossipEnvelope(akka://LeaderElectionSpec@localhost:36354,Gossip(overview = GossipOverview(seen = [akka://LeaderElectionSpec@localhost:59476 -> VectorClock(Node(ba2821b93a191a21dd71f0b8071762e1) -> 0000013a1f3ff5d4), akka://LeaderElectionSpec@localhost:36354 -> VectorClock(Node(ba2821b93a191a21dd71f0b8071762e1) -> 0000013a1f3ff5d4), akka://LeaderElectionSpec@localhost:41943 -> VectorClock(Node(ba2821b93a191a21dd71f0b8071762e1) -> 0000013a1f3ff5d4), akka://LeaderElectionSpec@localhost:46933 -> VectorClock(Node(ba2821b93a191a21dd71f0b8071762e1) -> 0000013a1f3ff5d4)], unreachable = []), members = [Member(address = akka://LeaderElectionSpec@localhost:36354, status = Joining), Member(address = akka://LeaderElectionSpec@localhost:41943, status = Joining), Member(address = akka://LeaderElectionSpec@localhost:46933, status = Joining), Member(address = akka://LeaderElectionSpec@localhost:59476, status = Up)], version = VectorClock(Node(ba2821b93a191a21dd71f0b8071762e1) -> 0000013a1f3ff5d4)),true)
[JVM-Node3] [WARN] [10/02/2012 04:13:36.928] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [akka://LeaderElectionSpec/system/FailureInjector] installing context org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@228ca8ac instead of org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@4095c5ec for address akka://LeaderElectionSpec@localhost:41943
[JVM-Node5] - must be able to 'elect' a single leader (1 second, 726 milliseconds)
[JVM-Node2] - must be able to 'elect' a single leader (1 second, 731 milliseconds)
[JVM-Node1] - must be able to 'elect' a single leader (1 second, 829 milliseconds)
[JVM-Node4] - must be able to 'elect' a single leader (1 second, 743 milliseconds)
[JVM-Node3] - must be able to 'elect' a single leader (1 second, 727 milliseconds)
[JVM-Node2] [INFO] [10/02/2012 04:13:37.660] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [akka://LeaderElectionSpec/system/cluster/core] Cluster Node [akka://LeaderElectionSpec@localhost:59476] - Marking node [akka://LeaderElectionSpec@localhost:36354] as DOWN
[JVM-Node1] [ERROR] [10/02/2012 04:13:37.666] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [akka://LeaderElectionSpec/user/controller/barriers] Actor[akka://LeaderElectionSpec/user/controller/127.0.0.1:37745-server2] tried to enter 'after-shutdown' while we were waiting for 'completed' (akka.remote.testconductor.BarrierCoordinator$WrongBarrier)
[JVM-Node1] - be able to 're-elect' a single leader after leader has left *** FAILED *** (38 milliseconds)
[JVM-Node2] - be able to 're-elect' a single leader after leader has left *** FAILED *** (39 milliseconds)
[JVM-Node2]   java.lang.RuntimeException: barrier failed: completed
[JVM-Node2]   at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:203)
[JVM-Node2]   at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:179)
[JVM-Node2]   at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:37)
[JVM-Node2]   at akka.actor.FSM$class.processEvent(FSM.scala:571)
[JVM-Node2]   at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:142)
[JVM-Node2]   at akka.actor.LoggingFSM$class.processEvent(FSM.scala:702)
[JVM-Node2]   at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:142)
[JVM-Node2]   at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:565)
[JVM-Node2]   at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:559)
[JVM-Node2]   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:394)
[JVM-Node2]   at akka.actor.ActorCell.invoke(ActorCell.scala:369)
[JVM-Node2]   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:230)
[JVM-Node2]   at akka.dispatch.Mailbox.run(Mailbox.scala:212)
[JVM-Node2]   at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:509)
[JVM-Node2]   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
[JVM-Node2]   at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
[JVM-Node2]   at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
[JVM-Node2]   at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
[JVM-Node1]   java.lang.RuntimeException: barrier failed: completed
[JVM-Node1]   at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:203)
[JVM-Node1]   at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:179)
[JVM-Node1]   at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:37)
[JVM-Node1]   at akka.actor.FSM$class.processEvent(FSM.scala:571)
[JVM-Node1]   at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:142)
[JVM-Node1]   at akka.actor.LoggingFSM$class.processEvent(FSM.scala:702)
[JVM-Node1]   at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:142)
[JVM-Node1]   at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:565)
[JVM-Node1]   at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:559)
[JVM-Node1]   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:394)
[JVM-Node1]   at akka.actor.ActorCell.invoke(ActorCell.scala:369)
[JVM-Node1]   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:230)
[JVM-Node1]   at akka.dispatch.Mailbox.run(Mailbox.scala:212)
[JVM-Node1]   at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:509)
[JVM-Node1]   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
[JVM-Node1]   at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
[JVM-Node1]   at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
[JVM-Node1]   at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
[JVM-Node1] [ERROR] [10/02/2012 04:13:37.681] [LeaderElectionSpec-akka.actor.default-dispatcher-6] [akka://LeaderElectionSpec/user/controller/barriers] cannot remove RoleName(controller): no client to remove (akka.remote.testconductor.BarrierCoordinator$BarrierEmpty)
[JVM-Node1] - be able to 're-elect' a single leader after leader has left (again) *** FAILED *** (5 milliseconds)
[JVM-Node1]   Previous step failed (MultiNodeClusterSpec.scala:82)
[JVM-Node1]   org.scalatest.exceptions.TestFailedException:
[JVM-Node1]   at akka.cluster.MultiNodeClusterSpec$class.withFixture(MultiNodeClusterSpec.scala:82)
[JVM-Node2] - be able to 're-elect' a single leader after leader has left (again) *** FAILED *** (4 milliseconds)
[JVM-Node2]   Previous step failed (MultiNodeClusterSpec.scala:82)
[JVM-Node2]   org.scalatest.exceptions.TestFailedException:
[JVM-Node2]   at akka.cluster.MultiNodeClusterSpec$class.withFixture(MultiNodeClusterSpec.scala:82)