LeaderElectionSpec failed
Leave a comment
on 2012-08-30 10:27 *
By Patrik Nordwall
I don't understand how the barriers can be wrong:
Full log:
[controller] [ERROR] [08/29/2012 08:44:01.437] [LeaderElectionSpec-akka.actor.default-dispatcher-3] [akka://LeaderElectionSpec/user/controller/barriers] Actor[akka://LeaderElectionSpec/user/controller/172.16.42.101:47022-server5] tried to enter 'after-shutdown' while we were waiting for 'after-down' (akka.remote.testconductor.BarrierCoordinator$WrongBarrier)
Full log:
[info] * akka.cluster.LeaderElectionWithFailureDetectorPuppet
[first] [INFO] [08/29/2012 08:43:58.248] [main] [NettyRemoteTransport(akka://LeaderElectionSpec@scalable2-01:38975)] RemoteServerStarted@akka://LeaderElectionSpec@scalable2-01:38975
[third] [INFO] [08/29/2012 08:43:58.265] [main] [NettyRemoteTransport(akka://LeaderElectionSpec@scalable2-03:48518)] RemoteServerStarted@akka://LeaderElectionSpec@scalable2-03:48518
[second] [INFO] [08/29/2012 08:43:58.319] [main] [NettyRemoteTransport(akka://LeaderElectionSpec@scalable2-02:42536)] RemoteServerStarted@akka://LeaderElectionSpec@scalable2-02:42536
[fourth] [INFO] [08/29/2012 08:43:58.349] [main] [NettyRemoteTransport(akka://LeaderElectionSpec@scalable2-04:43078)] RemoteServerStarted@akka://LeaderElectionSpec@scalable2-04:43078
[JVM-Node5] [INFO] [08/29/2012 08:43:58.443] [main] [NettyRemoteTransport(akka://LeaderElectionSpec@scalable2-01:41205)] RemoteServerStarted@akka://LeaderElectionSpec@scalable2-01:41205
[JVM-Node1] [INFO] [08/29/2012 08:43:58.656] [main] [LeaderElectionWithFailureDetectorPuppetMultiJvmNode1(akka://LeaderElectionSpec)] Role [controller] started with address [akka://LeaderElectionSpec@scalable2-01:38975]
[controller] LeaderElectionWithFailureDetectorPuppetMultiJvmNode1:
[controller] A cluster of four nodes
[JVM-Node2] [INFO] [08/29/2012 08:43:58.737] [main] [LeaderElectionWithFailureDetectorPuppetMultiJvmNode2(akka://LeaderElectionSpec)] Role [first] started with address [akka://LeaderElectionSpec@scalable2-02:42536]
[JVM-Node4] [INFO] [08/29/2012 08:43:58.744] [main] [LeaderElectionWithFailureDetectorPuppetMultiJvmNode4(akka://LeaderElectionSpec)] Role [third] started with address [akka://LeaderElectionSpec@scalable2-04:43078]
[JVM-Node3] [INFO] [08/29/2012 08:43:58.749] [main] [LeaderElectionWithFailureDetectorPuppetMultiJvmNode3(akka://LeaderElectionSpec)] Role [second] started with address [akka://LeaderElectionSpec@scalable2-03:48518]
[JVM-Node5] [INFO] [08/29/2012 08:43:58.764] [main] [LeaderElectionWithFailureDetectorPuppetMultiJvmNode5(akka://LeaderElectionSpec)] Role [fourth] started with address [akka://LeaderElectionSpec@scalable2-01:41205]
[third] LeaderElectionWithFailureDetectorPuppetMultiJvmNode4:
[third] A cluster of four nodes
[first] LeaderElectionWithFailureDetectorPuppetMultiJvmNode2:
[second] LeaderElectionWithFailureDetectorPuppetMultiJvmNode3:
[first] A cluster of four nodes
[fourth] LeaderElectionWithFailureDetectorPuppetMultiJvmNode5:
[fourth] A cluster of four nodes
[second] A cluster of four nodes
[first] [INFO] [08/29/2012 08:43:58.849] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@first] - is starting up...
[first] [INFO] [08/29/2012 08:43:58.850] [main] [Cluster(akka://LeaderElectionSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <= 'akka.cluster.scheduler.tick-duration' [33 ms].
[first] [INFO] [08/29/2012 08:43:58.911] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@first] - registered cluster JMX MBean [akka:type=Cluster]
[first] [INFO] [08/29/2012 08:43:58.912] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@first] - has started up successfully
[fourth] [INFO] [08/29/2012 08:43:58.953] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@fourth] - is starting up...
[third] [INFO] [08/29/2012 08:43:58.954] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@third] - is starting up...
[second] [INFO] [08/29/2012 08:43:58.953] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@second] - is starting up...
[fourth] [INFO] [08/29/2012 08:43:58.954] [main] [Cluster(akka://LeaderElectionSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <= 'akka.cluster.scheduler.tick-duration' [33 ms].
[second] [INFO] [08/29/2012 08:43:58.954] [main] [Cluster(akka://LeaderElectionSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <= 'akka.cluster.scheduler.tick-duration' [33 ms].
[third] [INFO] [08/29/2012 08:43:58.954] [main] [Cluster(akka://LeaderElectionSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <= 'akka.cluster.scheduler.tick-duration' [33 ms].
[third] [INFO] [08/29/2012 08:43:59.018] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@third] - registered cluster JMX MBean [akka:type=Cluster]
[third] [INFO] [08/29/2012 08:43:59.018] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@third] - has started up successfully
[second] [INFO] [08/29/2012 08:43:59.021] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@second] - registered cluster JMX MBean [akka:type=Cluster]
[fourth] [INFO] [08/29/2012 08:43:59.021] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@fourth] - registered cluster JMX MBean [akka:type=Cluster]
[second] [INFO] [08/29/2012 08:43:59.021] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@second] - has started up successfully
[fourth] [INFO] [08/29/2012 08:43:59.022] [main] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@fourth] - has started up successfully
[third] [INFO] [08/29/2012 08:43:59.077] [LeaderElectionSpec-akka.actor.default-dispatcher-2] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientStarted@akka://LeaderElectionSpec@first
[fourth] [INFO] [08/29/2012 08:43:59.081] [LeaderElectionSpec-akka.actor.default-dispatcher-1] [NettyRemoteTransport(akka://LeaderElectionSpec@fourth)] RemoteClientStarted@akka://LeaderElectionSpec@first
[second] [INFO] [08/29/2012 08:43:59.086] [LeaderElectionSpec-akka.actor.default-dispatcher-2] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientStarted@akka://LeaderElectionSpec@first
[first] [INFO] [08/29/2012 08:43:59.247] [LeaderElectionSpec-5] [NettyRemoteTransport(akka://LeaderElectionSpec@first)] RemoteClientStarted@akka://LeaderElectionSpec@second
[first] [INFO] [08/29/2012 08:43:59.250] [LeaderElectionSpec-4] [NettyRemoteTransport(akka://LeaderElectionSpec@first)] RemoteClientStarted@akka://LeaderElectionSpec@third
[first] [INFO] [08/29/2012 08:43:59.250] [LeaderElectionSpec-2] [NettyRemoteTransport(akka://LeaderElectionSpec@first)] RemoteClientStarted@akka://LeaderElectionSpec@fourth
[first] [INFO] [08/29/2012 08:43:59.251] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [akka://LeaderElectionSpec/system/cluster/core] Cluster Node [akka://LeaderElectionSpec@first] - Leader is moving node [akka://LeaderElectionSpec@first] from JOINING to UP
[second] [INFO] [08/29/2012 08:43:59.455] [LeaderElectionSpec-akka.actor.default-dispatcher-1] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientStarted@akka://LeaderElectionSpec@third
[third] [INFO] [08/29/2012 08:43:59.457] [LeaderElectionSpec-akka.actor.default-dispatcher-2] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientStarted@akka://LeaderElectionSpec@second
[fourth] [INFO] [08/29/2012 08:43:59.456] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [NettyRemoteTransport(akka://LeaderElectionSpec@fourth)] RemoteClientStarted@akka://LeaderElectionSpec@third
[second] [INFO] [08/29/2012 08:43:59.460] [LeaderElectionSpec-akka.actor.default-dispatcher-6] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientStarted@akka://LeaderElectionSpec@fourth
[third] [INFO] [08/29/2012 08:43:59.462] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientStarted@akka://LeaderElectionSpec@fourth
[fourth] [INFO] [08/29/2012 08:43:59.464] [LeaderElectionSpec-akka.actor.default-dispatcher-5] [NettyRemoteTransport(akka://LeaderElectionSpec@fourth)] RemoteClientStarted@akka://LeaderElectionSpec@second
[third] [INFO] [08/29/2012 08:43:59.466] [LeaderElectionSpec-4] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientStarted@akka://LeaderElectionSpec@fourth
[third] [INFO] [08/29/2012 08:43:59.468] [LeaderElectionSpec-4] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientShutdown@akka://LeaderElectionSpec@fourth
[second] [INFO] [08/29/2012 08:43:59.470] [LeaderElectionSpec-2] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientStarted@akka://LeaderElectionSpec@third
[second] [INFO] [08/29/2012 08:43:59.471] [LeaderElectionSpec-2] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientShutdown@akka://LeaderElectionSpec@third
[fourth] [INFO] [08/29/2012 08:43:59.472] [LeaderElectionSpec-5] [NettyRemoteTransport(akka://LeaderElectionSpec@fourth)] RemoteClientStarted@akka://LeaderElectionSpec@second
[fourth] [INFO] [08/29/2012 08:43:59.474] [LeaderElectionSpec-5] [NettyRemoteTransport(akka://LeaderElectionSpec@fourth)] RemoteClientShutdown@akka://LeaderElectionSpec@second
[third] [INFO] [08/29/2012 08:43:59.478] [LeaderElectionSpec-2] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientStarted@akka://LeaderElectionSpec@second
[third] [INFO] [08/29/2012 08:43:59.479] [LeaderElectionSpec-2] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientShutdown@akka://LeaderElectionSpec@second
[second] [INFO] [08/29/2012 08:43:59.481] [LeaderElectionSpec-3] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientStarted@akka://LeaderElectionSpec@fourth
[second] [INFO] [08/29/2012 08:43:59.481] [LeaderElectionSpec-3] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientShutdown@akka://LeaderElectionSpec@fourth
[third] [INFO] [08/29/2012 08:43:59.482] [LeaderElectionSpec-2] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientShutdown@akka://LeaderElectionSpec@second
[third] [INFO] [08/29/2012 08:43:59.484] [LeaderElectionSpec-5] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientShutdown@akka://LeaderElectionSpec@fourth
[second] [INFO] [08/29/2012 08:43:59.485] [LeaderElectionSpec-2] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientShutdown@akka://LeaderElectionSpec@third
[second] [INFO] [08/29/2012 08:43:59.485] [LeaderElectionSpec-3] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientShutdown@akka://LeaderElectionSpec@fourth
[fourth] [INFO] [08/29/2012 08:43:59.487] [LeaderElectionSpec-4] [NettyRemoteTransport(akka://LeaderElectionSpec@fourth)] RemoteClientStarted@akka://LeaderElectionSpec@third
[fourth] [INFO] [08/29/2012 08:43:59.487] [LeaderElectionSpec-4] [NettyRemoteTransport(akka://LeaderElectionSpec@fourth)] RemoteClientShutdown@akka://LeaderElectionSpec@third
[fourth] [INFO] [08/29/2012 08:43:59.488] [LeaderElectionSpec-4] [NettyRemoteTransport(akka://LeaderElectionSpec@fourth)] RemoteClientShutdown@akka://LeaderElectionSpec@third
[fourth] [INFO] [08/29/2012 08:43:59.489] [LeaderElectionSpec-5] [NettyRemoteTransport(akka://LeaderElectionSpec@fourth)] RemoteClientShutdown@akka://LeaderElectionSpec@second
[second] [INFO] [08/29/2012 08:43:59.504] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientStarted@akka://LeaderElectionSpec@fourth
[fourth] [INFO] [08/29/2012 08:43:59.508] [LeaderElectionSpec-4] [NettyRemoteTransport(akka://LeaderElectionSpec@fourth)] RemoteClientStarted@akka://LeaderElectionSpec@second
[third] [INFO] [08/29/2012 08:43:59.510] [LeaderElectionSpec-akka.actor.default-dispatcher-6] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientStarted@akka://LeaderElectionSpec@second
[second] [INFO] [08/29/2012 08:43:59.512] [LeaderElectionSpec-5] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientStarted@akka://LeaderElectionSpec@third
[third] [INFO] [08/29/2012 08:43:59.705] [LeaderElectionSpec-akka.actor.default-dispatcher-8] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientStarted@akka://LeaderElectionSpec@fourth
[fourth] [INFO] [08/29/2012 08:43:59.716] [LeaderElectionSpec-4] [NettyRemoteTransport(akka://LeaderElectionSpec@fourth)] RemoteClientStarted@akka://LeaderElectionSpec@third
[fourth] [INFO] [08/29/2012 08:44:00.729] [LeaderElectionSpec-akka.actor.default-dispatcher-1] [akka://LeaderElectionSpec/system/cluster/core] Cluster Node [akka://LeaderElectionSpec@fourth] - Leader is moving node [akka://LeaderElectionSpec@fourth] from JOINING to UP
[fourth] [INFO] [08/29/2012 08:44:00.730] [LeaderElectionSpec-akka.actor.default-dispatcher-1] [akka://LeaderElectionSpec/system/cluster/core] Cluster Node [akka://LeaderElectionSpec@fourth] - Leader is moving node [akka://LeaderElectionSpec@second] from JOINING to UP
[fourth] [INFO] [08/29/2012 08:44:00.731] [LeaderElectionSpec-akka.actor.default-dispatcher-1] [akka://LeaderElectionSpec/system/cluster/core] Cluster Node [akka://LeaderElectionSpec@fourth] - Leader is moving node [akka://LeaderElectionSpec@third] from JOINING to UP
[fourth] - must be able to 'elect' a single leader (2 seconds, 555 milliseconds)
[third] - must be able to 'elect' a single leader (2 seconds, 583 milliseconds)
[second] - must be able to 'elect' a single leader (2 seconds, 549 milliseconds)
[controller] - must be able to 'elect' a single leader (2 seconds, 667 milliseconds)
[first] - must be able to 'elect' a single leader (2 seconds, 567 milliseconds)
[third] [INFO] [08/29/2012 08:44:01.434] [LeaderElectionSpec-akka.actor.default-dispatcher-5] [akka://LeaderElectionSpec/system/cluster/core] Cluster Node [akka://LeaderElectionSpec@third] - Marking node [akka://LeaderElectionSpec@fourth] as DOWN
[controller] [ERROR] [08/29/2012 08:44:01.437] [LeaderElectionSpec-akka.actor.default-dispatcher-3] [akka://LeaderElectionSpec/user/controller/barriers] Actor[akka://LeaderElectionSpec/user/controller/172.16.42.101:47022-server5] tried to enter 'after-shutdown' while we were waiting for 'after-down' (akka.remote.testconductor.BarrierCoordinator$WrongBarrier)
[third] - be able to 're-elect' a single leader after leader has left *** FAILED *** (43 milliseconds)
[third] java.lang.RuntimeException: barrier failed: after-down
[third] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:200)
[third] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:176)
[third] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:37)
[third] at akka.actor.FSM$class.processEvent(FSM.scala:568)
[third] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:139)
[third] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:699)
[third] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:139)
[third] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:562)
[third] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:556)
[third] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:388)
[third] at akka.actor.ActorCell.invoke(ActorCell.scala:364)
[third] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:230)
[third] at akka.dispatch.Mailbox.run(Mailbox.scala:212)
[third] at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
[third] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
[third] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
[third] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
[third] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
[first] - be able to 're-elect' a single leader after leader has left *** FAILED *** (35 milliseconds)
[first] java.lang.RuntimeException: barrier failed: after-down
[first] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:200)
[first] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:176)
[first] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:37)
[first] at akka.actor.FSM$class.processEvent(FSM.scala:568)
[first] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:139)
[first] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:699)
[first] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:139)
[first] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:562)
[first] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:556)
[first] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:388)
[first] at akka.actor.ActorCell.invoke(ActorCell.scala:364)
[first] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:230)
[first] at akka.dispatch.Mailbox.run(Mailbox.scala:212)
[first] at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
[first] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
[first] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
[first] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
[first] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
[third] - be able to 're-elect' a single leader after leader has left (again) *** FAILED *** (2 milliseconds)
[third] Previous step failed (MultiNodeClusterSpec.scala:71)
[third] org.scalatest.exceptions.TestFailedException:
[third] at akka.cluster.MultiNodeClusterSpec$class.withFixture(MultiNodeClusterSpec.scala:71)
[first] - be able to 're-elect' a single leader after leader has left (again) *** FAILED *** (3 milliseconds)
[first] Previous step failed (MultiNodeClusterSpec.scala:71)
[first] org.scalatest.exceptions.TestFailedException:
[first] at akka.cluster.MultiNodeClusterSpec$class.withFixture(MultiNodeClusterSpec.scala:71)
[controller] - be able to 're-elect' a single leader after leader has left *** FAILED *** (49 milliseconds)
[controller] java.lang.RuntimeException: barrier failed: after-down
[controller] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:200)
[controller] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:176)
[controller] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:37)
[controller] at akka.actor.FSM$class.processEvent(FSM.scala:568)
[controller] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:139)
[controller] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:699)
[controller] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:139)
[controller] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:562)
[controller] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:556)
[controller] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:388)
[controller] at akka.actor.ActorCell.invoke(ActorCell.scala:364)
[controller] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:230)
[controller] at akka.dispatch.Mailbox.run(Mailbox.scala:212)
[controller] at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
[controller] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
[controller] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
[controller] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
[controller] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
[third] [WARN] [08/29/2012 08:44:01.454] [LeaderElectionSpec-akka.actor.default-dispatcher-5] [akka://LeaderElectionSpec/user/TestConductorClient] received dead letter from Actor[akka://LeaderElectionSpec/deadLetters]: PoisonPill
[second] - be able to 're-elect' a single leader after leader has left *** FAILED *** (49 milliseconds)
[second] java.lang.RuntimeException: barrier failed: after-down
[second] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:200)
[second] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:176)
[second] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:37)
[second] at akka.actor.FSM$class.processEvent(FSM.scala:568)
[second] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:139)
[second] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:699)
[second] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:139)
[second] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:562)
[second] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:556)
[second] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:388)
[second] at akka.actor.ActorCell.invoke(ActorCell.scala:364)
[second] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:230)
[second] at akka.dispatch.Mailbox.run(Mailbox.scala:212)
[second] at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
[second] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
[second] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
[second] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
[second] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
[second] - be able to 're-elect' a single leader after leader has left (again) *** FAILED *** (5 milliseconds)
[second] Previous step failed (MultiNodeClusterSpec.scala:71)
[second] org.scalatest.exceptions.TestFailedException:
[second] at akka.cluster.MultiNodeClusterSpec$class.withFixture(MultiNodeClusterSpec.scala:71)
[controller] [ERROR] [08/29/2012 08:44:01.462] [LeaderElectionSpec-akka.actor.default-dispatcher-3] [akka://LeaderElectionSpec/user/controller/barriers] cannot remove RoleName(controller): no client to remove (akka.remote.testconductor.BarrierCoordinator$BarrierEmpty)
[first] [WARN] [08/29/2012 08:44:01.462] [LeaderElectionSpec-akka.actor.default-dispatcher-2] [akka://LeaderElectionSpec/user/TestConductorClient] received dead letter from Actor[akka://LeaderElectionSpec/deadLetters]: PoisonPill
[second] [WARN] [08/29/2012 08:44:01.462] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [akka://LeaderElectionSpec/user/TestConductorClient] received dead letter from Actor[akka://LeaderElectionSpec/deadLetters]: PoisonPill
[controller] - be able to 're-elect' a single leader after leader has left (again) *** FAILED *** (2 milliseconds)
[controller] Previous step failed (MultiNodeClusterSpec.scala:71)
[controller] org.scalatest.exceptions.TestFailedException:
[controller] at akka.cluster.MultiNodeClusterSpec$class.withFixture(MultiNodeClusterSpec.scala:71)
[third] [INFO] [08/29/2012 08:44:01.467] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@third] - Shutting down cluster Node and cluster daemons...
[third] [INFO] [08/29/2012 08:44:01.472] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@third] - Cluster node successfully shut down
[second] [INFO] [08/29/2012 08:44:01.476] [LeaderElectionSpec-akka.actor.default-dispatcher-6] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@second] - Shutting down cluster Node and cluster daemons...
[first] [INFO] [08/29/2012 08:44:01.477] [LeaderElectionSpec-akka.actor.default-dispatcher-8] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@first] - Shutting down cluster Node and cluster daemons...
[third] [INFO] [08/29/2012 08:44:01.477] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientShutdown@akka://LeaderElectionSpec@fourth
[third] [INFO] [08/29/2012 08:44:01.478] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientShutdown@akka://LeaderElectionSpec@second
[third] [INFO] [08/29/2012 08:44:01.479] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteClientShutdown@akka://LeaderElectionSpec@first
[second] [INFO] [08/29/2012 08:44:01.480] [LeaderElectionSpec-2] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientShutdown@akka://LeaderElectionSpec@third
[second] [INFO] [08/29/2012 08:44:01.482] [LeaderElectionSpec-akka.actor.default-dispatcher-6] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@second] - Cluster node successfully shut down
[first] [INFO] [08/29/2012 08:44:01.482] [LeaderElectionSpec-akka.actor.default-dispatcher-8] [Cluster(akka://LeaderElectionSpec)] Cluster Node [akka://LeaderElectionSpec@first] - Cluster node successfully shut down
[third] [INFO] [08/29/2012 08:44:01.482] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [NettyRemoteTransport(akka://LeaderElectionSpec@third)] RemoteServerShutdown@akka://LeaderElectionSpec@third
[second] [INFO] [08/29/2012 08:44:01.483] [LeaderElectionSpec-akka.actor.default-dispatcher-6] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientShutdown@akka://LeaderElectionSpec@fourth
[second] [INFO] [08/29/2012 08:44:01.484] [LeaderElectionSpec-akka.actor.default-dispatcher-6] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteClientShutdown@akka://LeaderElectionSpec@first
[first] [INFO] [08/29/2012 08:44:01.484] [LeaderElectionSpec-2] [NettyRemoteTransport(akka://LeaderElectionSpec@first)] RemoteClientShutdown@akka://LeaderElectionSpec@third
[first] [INFO] [08/29/2012 08:44:01.487] [LeaderElectionSpec-3] [NettyRemoteTransport(akka://LeaderElectionSpec@first)] RemoteClientShutdown@akka://LeaderElectionSpec@second
[second] [INFO] [08/29/2012 08:44:01.488] [LeaderElectionSpec-akka.actor.default-dispatcher-6] [NettyRemoteTransport(akka://LeaderElectionSpec@second)] RemoteServerShutdown@akka://LeaderElectionSpec@second
[first] [INFO] [08/29/2012 08:44:01.488] [LeaderElectionSpec-akka.actor.default-dispatcher-8] [NettyRemoteTransport(akka://LeaderElectionSpec@first)] RemoteClientShutdown@akka://LeaderElectionSpec@fourth
[controller] Aug 29, 2012 8:44:01 AM org.jboss.netty.channel.SimpleChannelUpstreamHandler
[controller] WARNING: EXCEPTION, please implement akka.remote.testconductor.ConductorHandler.exceptionCaught() for proper handling.
[controller] java.io.IOException: Connection reset by peer
[controller] at sun.nio.ch.FileDispatcher.read0(Native Method)
[controller] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
[controller] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)
[controller] at sun.nio.ch.IOUtil.read(IOUtil.java:218)
[controller] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
[controller] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:69)
[controller] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:390)
[controller] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
[controller] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
[controller] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[controller] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[controller] at java.lang.Thread.run(Thread.java:679)
[controller] [WARN] [08/29/2012 08:44:01.785] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [akka://LeaderElectionSpec/user/TestConductorClient] received dead letter from Actor[akka://LeaderElectionSpec/deadLetters]: PoisonPill
[controller] [WARN] [08/29/2012 08:44:01.785] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [akka://LeaderElectionSpec/user/controller/172.16.42.101:47020-server2] received dead letter from Actor[akka://LeaderElectionSpec/deadLetters]: ClientDisconnected
[controller] [WARN] [08/29/2012 08:44:01.786] [LeaderElectionSpec-akka.actor.default-dispatcher-7] [akka://LeaderElectionSpec/user/controller] received dead letter from Actor[akka://LeaderElectionSpec/user/controller/172.16.42.101:47020-server2]: ClientDisconnected(RoleName(controller))
[controller] [INFO] [08/29/2012 08:44:01.818] [LeaderElectionSpec-akka.actor.default-dispatcher-5] [NettyRemoteTransport(akka://LeaderElectionSpec@controller)] RemoteServerShutdown@akka://LeaderElectionSpec@controller
[first] [WARN] [08/29/2012 08:44:06.454] [main] [ActorSystem(LeaderElectionSpec)] Failed to stop [LeaderElectionSpec] within 5 seconds
[first] -> / LocalActorRefProvider$$anon$1 null status=1 terminated
[error] Failed: akka.cluster.LeaderElectionWithFailureDetectorPuppetMultiJvmNode1
[error] Failed: akka.cluster.LeaderElectionWithFailureDetectorPuppetMultiJvmNode2
[error] Failed: akka.cluster.LeaderElectionWithFailureDetectorPuppetMultiJvmNode3
[error] Failed: akka.cluster.LeaderElectionWithFailureDetectorPuppetMultiJvmNode4
on 2012-09-14 03:45 *
By Patrik Nordwall
Something fishy with this test.
cannot remove RoleName(controller): no client to remove (akka.remote.testconductor.BarrierCoordinator$BarrierEmpty)
http://scalable1:8080/job/akka-multi-node-nightly/124/consoleText
cannot remove RoleName(controller): no client to remove (akka.remote.testconductor.BarrierCoordinator$BarrierEmpty)
http://scalable1:8080/job/akka-multi-node-nightly/124/consoleText
[JVM-Node1] [ERROR] [09/14/2012 08:16:55.542] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [akka://LeaderElectionSpec/user/controller/barriers] unannounced disconnect of RoleName(second) (akka.remote.testconductor.BarrierCoordinator$ClientLost)
[JVM-Node1] [ERROR] [09/14/2012 08:16:55.550] [LeaderElectionSpec-akka.actor.default-dispatcher-4] [akka://LeaderElectionSpec/user/controller/barriers] cannot remove RoleName(controller): no client to remove (akka.remote.testconductor.BarrierCoordinator$BarrierEmpty)
[JVM-Node1] [31m- be able to 're-elect' a single leader after leader has left (again) *** FAILED ***[0m
[JVM-Node1] [31m java.lang.RuntimeException: barrier failed: completed[0m
[JVM-Node1] [31m at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:203)[0m
[JVM-Node1] [31m at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:179)[0m
[JVM-Node1] [31m at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:37)[0m
[JVM-Node1] [31m at akka.actor.FSM$class.processEvent(FSM.scala:568)[0m
[JVM-Node1] [31m at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:142)[0m
[JVM-Node1] [31m at akka.actor.LoggingFSM$class.processEvent(FSM.scala:699)[0m
[JVM-Node1] [31m at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:142)[0m
[JVM-Node1] [31m at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:562)[0m
[JVM-Node1] [31m at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:556)[0m
[JVM-Node1] [31m at akka.actor.ActorCell.receiveMessage(ActorCell.scala:392)[0m
on 2012-09-24 08:11 *
By Patrik Nordwall
Assigned to changed from Patrik Nordwall to -none-
Priority changed from High (2) to Normal (3)
Status changed from Accepted to Invalid
I tried hard to understand how the barriers could come in wrong order, but I'm not able to see anything wrong with the test. Closing it as not possible to reproduce.
Updating tickets (#939, #940, #1941, #2213, #2214, #2215, #2219, #2222, #2223, #2239, #2240, #2249, #2250, #2252, #2253, #2254, #2256, #2259, #2263, #2264, #2265, #2267, #2270, #2271, #2275, #2277, #2286, #2287, #2289, #2290, #2303, #2304, #2308, #2310, #2311, #2317, #2323, #2331, #2374, #2392, #2405, #2423, #2425, #2440, #2444, #2445, #2453, #2456, #2459, #2473, #2477, #2491, #2495, #2523, #2534, #2541, #2544, #2545, #2549, #2582, #2583, #2589, #2626)
Updating tickets (#939, #940, #1941, #2081, #2126, #2213, #2214, #2215, #2219, #2222, #2223, #2239, #2240, #2249, #2250, #2252, #2253, #2254, #2256, #2259, #2263, #2264, #2265, #2267, #2270, #2271, #2275, #2277, #2286, #2287, #2289, #2290, #2303, #2304, #2308, #2310, #2311, #2317, #2323, #2331, #2374, #2392, #2394, #2405, #2408, #2423, #2424, #2425, #2440, #2444, #2445, #2449, #2453, #2456, #2459, #2461, #2473, #2477, #2485, #2491, #2495, #2498, #2501, #2505, #2515, #2517, #2523, #2534, #2541, #2544, #2545, #2549, #2582, #2583, #2588, #2589, #2598, #2599, #2618, #2623, #2626, #2627, #2630, #2631, #2633, #2634, #2635, #2637, #2638, #2642, #2643, #2646, #2647, #2648, #2649, #2650, #2653, #2655, #2657, #2658)