#2654
- Created on2012-10-25 03:22
- Reported by
- StatusFixed
- PriorityHighest (1)
- Assigned to
- Milestone
- Plan Level-
- Due Date-
Unexpected deadletter in StatsSampleSpec
akka-multi-node/33 at jenkins.akka.io
[info] * sample.cluster.stats.StatsSampleSpec
[JVM-Node1] Run starting. Expected test count is: 3
[JVM-Node1] StatsSampleSpecMultiJvmNode1:
[JVM-Node1] The stats sample
[JVM-Node2] Run starting. Expected test count is: 3
[JVM-Node3] Run starting. Expected test count is: 3
[JVM-Node2] StatsSampleSpecMultiJvmNode2:
[JVM-Node3] StatsSampleSpecMultiJvmNode3:
[JVM-Node2] The stats sample
[JVM-Node3] The stats sample
[JVM-Node1] - must illustrate how to startup cluster
[JVM-Node3] - must illustrate how to startup cluster
[JVM-Node2] - must illustrate how to startup cluster
[JVM-Node2] [WARN] [10/25/2012 12:07:36.875] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a5.local:40973/user/statsService] received dead letter from Actor[akka://StatsSampleSpec/system/testActor1]: StatsJob(this is the text that will be analyzed)
[JVM-Node2] - show usage of the statsService from one node *** FAILED ***
[JVM-Node2] java.lang.AssertionError: assertion failed: timeout (4982698809 nanoseconds) during expectMsg:
[JVM-Node2] at scala.Predef$.assert(Predef.scala:179)
[JVM-Node2] at akka.testkit.TestKitBase$class.expectMsgPF(TestKit.scala:312)
[JVM-Node2] at akka.testkit.TestKit.expectMsgPF(TestKit.scala:641)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4$$anonfun$apply$mcV$sp$5.apply$mcV$sp(StatsSampleSpec.scala:113)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec.runOn(MultiNodeSpec.scala:322)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4.apply$mcV$sp(StatsSampleSpec.scala:110)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4.apply(StatsSampleSpec.scala:109)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4.apply(StatsSampleSpec.scala:109)
[JVM-Node2] at akka.testkit.TestKitBase$class.within(TestKit.scala:262)
[JVM-Node2] at akka.testkit.TestKit.within(TestKit.scala:641)
[JVM-Node2] ...
[JVM-Node1] [ERROR] [10/25/2012 12:07:41.903] [StatsSampleSpec-akka.actor.default-dispatcher-5] [akka://StatsSampleSpec/user/controller/barriers] RoleName(second) tried to enter 'done-3' while we were waiting for 'done-2' (akka.remote.testconductor.BarrierCoordinator$WrongBarrier)
[JVM-Node2] - show usage of the statsService from all nodes *** FAILED ***
[JVM-Node2] java.lang.RuntimeException: barrier failed: done-3
[JVM-Node2] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node2] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node2] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node2] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node2] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node2] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node2] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node2] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node2] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node2] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node2] ...
[JVM-Node3] - show usage of the statsService from one node *** FAILED ***
[JVM-Node3] java.lang.RuntimeException: barrier failed: done-2
[JVM-Node3] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node3] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node3] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node3] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node3] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node3] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node3] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node3] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node3] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node3] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node3] ...
[JVM-Node2] [WARN] [10/25/2012 12:07:41.914] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec/user/TestConductorClient] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: PoisonPill
[JVM-Node2] [WARN] [10/25/2012 12:07:41.916] [StatsSampleSpec-akka.actor.default-dispatcher-5] [akka://StatsSampleSpec/user/statsWorker] received dead letter from Actor[akka://StatsSampleSpec@a5.local:40973/user/statsService/$b]: the
[JVM-Node1] - show usage of the statsService from one node *** FAILED ***
[JVM-Node1] java.lang.RuntimeException: barrier failed: done-2
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node1] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node1] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node1] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node1] ...
[JVM-Node2] Run completed in 7 seconds, 566 milliseconds.
[JVM-Node2] Total number of tests run: 3
[JVM-Node2] Suites: completed 1, aborted 0
[JVM-Node2] Tests: succeeded 1, failed 2, ignored 0, pending 0
[JVM-Node2] *** 2 TESTS FAILED ***
[JVM-Node1] - show usage of the statsService from all nodes *** FAILED ***
[JVM-Node1] java.lang.RuntimeException: barrier failed: done-3
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node1] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node1] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node1] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node1] ...
[JVM-Node1] [ERROR] [10/25/2012 12:07:41.985] [StatsSampleSpec-akka.actor.default-dispatcher-4] [akka://StatsSampleSpec/user/controller/barriers] cannot remove RoleName(first): no client to remove (akka.remote.testconductor.BarrierCoordinator$BarrierEmpty)
[JVM-Node1] [WARN] [10/25/2012 12:07:42.751] [StatsSampleSpec-akka.actor.default-dispatcher-4] [akka://StatsSampleSpec@a4.local:44787/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A44787]: Heartbeat(akka://StatsSampleSpec@a3.local:36810)
[JVM-Node3] [WARN] [10/25/2012 12:07:42.829] [StatsSampleSpec-akka.actor.default-dispatcher-7] [akka://StatsSampleSpec@a4.local:44787/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A44787]: Heartbeat(akka://StatsSampleSpec@a5.local:40973)
[JVM-Node3] [WARN] [10/25/2012 12:07:42.833] [StatsSampleSpec-akka.actor.default-dispatcher-7] [akka://StatsSampleSpec@a4.local:44787/system/cluster/core] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/coreSender]: GossipEnvelope(akka://StatsSampleSpec@a5.local:40973,Gossip(overview = GossipOverview(seen = [akka://StatsSampleSpec@a3.local:36810 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a5.local:40973 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a4.local:44787 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)], unreachable = []), members = [Member(address = akka://StatsSampleSpec@a3.local:36810, status = Up), Member(address = akka://StatsSampleSpec@a4.local:44787, status = Up), Member(address = akka://StatsSampleSpec@a5.local:40973, status = Up)], version = VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)),true)
[JVM-Node1] [WARN] [10/25/2012 12:07:43.729] [StatsSampleSpec-akka.actor.default-dispatcher-3] [akka://StatsSampleSpec@a4.local:44787/system/cluster/core] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/coreSender]: GossipEnvelope(akka://StatsSampleSpec@a3.local:36810,Gossip(overview = GossipOverview(seen = [akka://StatsSampleSpec@a3.local:36810 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a5.local:40973 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a4.local:44787 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)], unreachable = []), members = [Member(address = akka://StatsSampleSpec@a3.local:36810, status = Up), Member(address = akka://StatsSampleSpec@a4.local:44787, status = Up), Member(address = akka://StatsSampleSpec@a5.local:40973, status = Up)], version = VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)),true)
[JVM-Node1] [WARN] [10/25/2012 12:07:43.758] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:44787/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A44787]: Heartbeat(akka://StatsSampleSpec@a3.local:36810)
[JVM-Node3] [WARN] [10/25/2012 12:07:43.817] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:44787/system/cluster/core] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/coreSender]: GossipEnvelope(akka://StatsSampleSpec@a5.local:40973,Gossip(overview = GossipOverview(seen = [akka://StatsSampleSpec@a3.local:36810 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a5.local:40973 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a4.local:44787 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)], unreachable = []), members = [Member(address = akka://StatsSampleSpec@a3.local:36810, status = Up), Member(address = akka://StatsSampleSpec@a4.local:44787, status = Up), Member(address = akka://StatsSampleSpec@a5.local:40973, status = Up)], version = VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)),true)
[JVM-Node3] [WARN] [10/25/2012 12:07:43.846] [StatsSampleSpec-akka.actor.default-dispatcher-7] [akka://StatsSampleSpec@a4.local:44787/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A44787]: Heartbeat(akka://StatsSampleSpec@a5.local:40973)
[JVM-Node1] [WARN] [10/25/2012 12:07:44.722] [StatsSampleSpec-akka.actor.default-dispatcher-5] [akka://StatsSampleSpec@a4.local:44787/system/cluster/metrics] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/metrics]: MetricsGossipEnvelope(akka://StatsSampleSpec@a3.local:36810,MetricsGossip(10,Set(NodeMetrics(akka://StatsSampleSpec@a3.local:36810,1351159661710,Set(Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,1,1351159658717,1351159661713))), Metric(system-load-average,Some(0.25),None), Metric(network-max-tx,None,None), Metric(processors,Some(16),None), Metric(heap-memory-used,Some(58525024),Some(DataStream(10,2266657,1351159658717,1351159661713))), Metric(total-cores,None,None), Metric(cpu-combined,None,None), Metric(network-max-rx,None,None), Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,1,1351159658717,1351159661713))))), NodeMetrics(akka://StatsSampleSpec@a5.local:40973,1351159658793,Set(Metric(system-load-average,Some(0.0),None), Metric(network-max-tx,None,None), Metric(processors,Some(16),None), Metric(total-cores,None,None), Metric(heap-memory-used,Some(56906256),Some(DataStream(10,56906256,1351159658805,1351159658805))), Metric(cpu-combined,None,None), Metric(network-max-rx,None,None), Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,7497580544,1351159658806,1351159658806))), Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,505217024,1351159658806,1351159658806))))), NodeMetrics(akka://StatsSampleSpec@a4.local:44787,1351159658802,Set(Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,7497580544,1351159658812,1351159658812))), Metric(network-max-tx,None,None), Metric(processors,Some(16),None), Metric(total-cores,None,None), Metric(heap-memory-used,Some(58988864),Some(DataStream(10,58988864,1351159658813,1351159658813))), Metric(cpu-combined,None,None), Metric(network-max-rx,None,None), Metric(system-load-average,Some(0.47),None), Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,505217024,1351159658812,1351159658812))))))))
[JVM-Node1] [WARN] [10/25/2012 12:07:44.748] [StatsSampleSpec-akka.actor.default-dispatcher-7] [akka://StatsSampleSpec@a4.local:44787/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A44787]: Heartbeat(akka://StatsSampleSpec@a3.local:36810)
[JVM-Node3] [WARN] [10/25/2012 12:07:44.811] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:44787/system/cluster/metrics] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/metrics]: MetricsGossipEnvelope(akka://StatsSampleSpec@a5.local:40973,MetricsGossip(10,Set(NodeMetrics(akka://StatsSampleSpec@a3.local:36810,1351159661710,Set(Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,1,1351159658717,1351159661713))), Metric(system-load-average,Some(0.25),None), Metric(network-max-tx,None,None), Metric(processors,Some(16),None), Metric(heap-memory-used,Some(58525024),Some(DataStream(10,2266657,1351159658717,1351159661713))), Metric(total-cores,None,None), Metric(cpu-combined,None,None), Metric(network-max-rx,None,None), Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,1,1351159658717,1351159661713))))), NodeMetrics(akka://StatsSampleSpec@a4.local:44787,1351159661800,Set(Metric(heap-memory-used,Some(60059304),Some(DataStream(10,1070441,1351159658813,1351159661803))), Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,1,1351159658812,1351159661803))), Metric(network-max-tx,None,None), Metric(processors,Some(16),None), Metric(total-cores,None,None), Metric(cpu-combined,None,None), Metric(network-max-rx,None,None), Metric(system-load-average,Some(0.44),None), Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,1,1351159658812,1351159661803))))), NodeMetrics(akka://StatsSampleSpec@a5.local:40973,1351159664799,Set(Metric(heap-memory-used,Some(65376368),Some(DataStream(10,62940056,1351159658805,1351159664800))), Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,505217024,1351159658806,1351159664800))), Metric(system-load-average,Some(0.0),None), Metric(network-max-tx,None,None), Metric(processors,Some(16),None), Metric(total-cores,None,None), Metric(cpu-combined,None,None), Metric(network-max-rx,None,None), Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,7497580544,1351159658806,1351159664800))))))))
[JVM-Node3] [WARN] [10/25/2012 12:07:44.837] [StatsSampleSpec-akka.actor.default-dispatcher-3] [akka://StatsSampleSpec@a4.local:44787/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A44787]: Heartbeat(akka://StatsSampleSpec@a5.local:40973)
[JVM-Node3] [WARN] [10/25/2012 12:07:44.840] [StatsSampleSpec-akka.actor.default-dispatcher-3] [akka://StatsSampleSpec@a4.local:44787/system/cluster/core] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/coreSender]: GossipEnvelope(akka://StatsSampleSpec@a5.local:40973,Gossip(overview = GossipOverview(seen = [akka://StatsSampleSpec@a3.local:36810 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a5.local:40973 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a4.local:44787 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)], unreachable = []), members = [Member(address = akka://StatsSampleSpec@a3.local:36810, status = Up), Member(address = akka://StatsSampleSpec@a4.local:44787, status = Up), Member(address = akka://StatsSampleSpec@a5.local:40973, status = Up)], version = VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)),true)
[JVM-Node1] [WARN] [10/25/2012 12:07:45.739] [StatsSampleSpec-akka.actor.default-dispatcher-5] [akka://StatsSampleSpec@a4.local:44787/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A44787]: Heartbeat(akka://StatsSampleSpec@a3.local:36810)
[JVM-Node1] [WARN] [10/25/2012 12:07:45.741] [StatsSampleSpec-akka.actor.default-dispatcher-5] [akka://StatsSampleSpec@a4.local:44787/system/cluster/core] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/coreSender]: GossipEnvelope(akka://StatsSampleSpec@a3.local:36810,Gossip(overview = GossipOverview(seen = [akka://StatsSampleSpec@a3.local:36810 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a5.local:40973 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a4.local:44787 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)], unreachable = []), members = [Member(address = akka://StatsSampleSpec@a3.local:36810, status = Up), Member(address = akka://StatsSampleSpec@a4.local:44787, status = Up), Member(address = akka://StatsSampleSpec@a5.local:40973, status = Up)], version = VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)),true)
[JVM-Node3] [WARN] [10/25/2012 12:07:45.828] [StatsSampleSpec-akka.actor.default-dispatcher-4] [akka://StatsSampleSpec@a4.local:44787/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A44787]: Heartbeat(akka://StatsSampleSpec@a5.local:40973)
[JVM-Node1] [ERROR] [10/25/2012 12:07:46.727] [StatsSampleSpec-akka.actor.default-dispatcher-7] [akka://StatsSampleSpec/system/cluster/core] Cluster Node [akka://StatsSampleSpec@a3.local:36810] - Marking node(s) as UNREACHABLE [Member(address = akka://StatsSampleSpec@a4.local:44787, status = Up)]
[JVM-Node1] [WARN] [10/25/2012 12:07:46.761] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:44787/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A44787]: EndHeartbeat(akka://StatsSampleSpec@a3.local:36810)
[JVM-Node3] [ERROR] [10/25/2012 12:07:46.818] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec/system/cluster/core] Cluster Node [akka://StatsSampleSpec@a5.local:40973] - Marking node(s) as UNREACHABLE [Member(address = akka://StatsSampleSpec@a4.local:44787, status = Up)]
[JVM-Node3] [WARN] [10/25/2012 12:07:46.821] [StatsSampleSpec-akka.actor.default-dispatcher-4] [akka://StatsSampleSpec@a4.local:44787/system/cluster/core] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/coreSender]: GossipEnvelope(akka://StatsSampleSpec@a5.local:40973,Gossip(overview = GossipOverview(seen = [akka://StatsSampleSpec@a3.local:36810 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a5.local:40973 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d), akka://StatsSampleSpec@a4.local:44787 -> VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)], unreachable = []), members = [Member(address = akka://StatsSampleSpec@a3.local:36810, status = Up), Member(address = akka://StatsSampleSpec@a4.local:44787, status = Up), Member(address = akka://StatsSampleSpec@a5.local:40973, status = Up)], version = VectorClock(Node(482e66eef5cc7a825f165ab9dae5352a) -> 0000013a9764311d)),true)
[JVM-Node3] [WARN] [10/25/2012 12:07:46.851] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec@a4.local:44787/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A44787]: EndHeartbeat(akka://StatsSampleSpec@a5.local:40973)
[JVM-Node3] - show usage of the statsService from all nodes *** FAILED ***
[JVM-Node3] java.lang.AssertionError: assertion failed: timeout (4996908116 nanoseconds) during expectMsg:
[JVM-Node3] at scala.Predef$.assert(Predef.scala:179)
[JVM-Node3] at akka.testkit.TestKitBase$class.expectMsgPF(TestKit.scala:312)
[JVM-Node3] at akka.testkit.TestKit.expectMsgPF(TestKit.scala:641)
[JVM-Node3] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$6$$anonfun$apply$mcV$sp$7.apply$mcV$sp(StatsSampleSpec.scala:126)
[JVM-Node3] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$6$$anonfun$apply$mcV$sp$7.apply(StatsSampleSpec.scala:123)
[JVM-Node3] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$6$$anonfun$apply$mcV$sp$7.apply(StatsSampleSpec.scala:123)
[JVM-Node3] at akka.testkit.TestKitBase$class.within(TestKit.scala:262)
[JVM-Node3] at akka.testkit.TestKit.within(TestKit.scala:641)
[JVM-Node3] at akka.testkit.TestKitBase$class.within(TestKit.scala:276)
[JVM-Node3] at akka.testkit.TestKit.within(TestKit.scala:641)
[JVM-Node3] ...
[JVM-Node3] [WARN] [10/25/2012 12:07:46.921] [StatsSampleSpec-akka.actor.default-dispatcher-5] [akka://StatsSampleSpec/user/TestConductorClient] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: PoisonPill
[JVM-Node3] Run completed in 12 seconds, 371 milliseconds.
[JVM-Node3] Total number of tests run: 3
[JVM-Node3] Suites: completed 1, aborted 0
[JVM-Node3] Tests: succeeded 1, failed 2, ignored 0, pending 0
[JVM-Node3] *** 2 TESTS FAILED ***
[JVM-Node1] [WARN] [10/25/2012 12:07:47.010] [StatsSampleSpec-akka.actor.default-dispatcher-8] [akka://StatsSampleSpec/user/controller/10.0.1.233:54143-server1] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: ClientDisconnected
[JVM-Node1] [WARN] [10/25/2012 12:07:47.012] [StatsSampleSpec-akka.actor.default-dispatcher-7] [akka://StatsSampleSpec/user/TestConductorClient] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: PoisonPill
[JVM-Node1] [WARN] [10/25/2012 12:07:47.012] [StatsSampleSpec-akka.actor.default-dispatcher-7] [akka://StatsSampleSpec/user/controller] received dead letter from Actor[akka://StatsSampleSpec/user/controller/10.0.1.233:54143-server1]: ClientDisconnected(RoleName(first))
[JVM-Node1] Run completed in 12 seconds, 849 milliseconds.
[JVM-Node1] Total number of tests run: 3
[JVM-Node1] Suites: completed 1, aborted 0
[JVM-Node1] Tests: succeeded 1, failed 2, ignored 0, pending 0
[JVM-Node1] *** 2 TESTS FAILED ***
[error] Failed: sample.cluster.stats.StatsSampleSpecMultiJvmNode1
[error] Failed: sample.cluster.stats.StatsSampleSpecMultiJvmNode2
[error] Failed: sample.cluster.stats.StatsSampleSpecMultiJvmNode3
Leave a comment
on 2012-10-30 04:32 *
By viktorklang
Seem to have happened again:
[info] * sample.cluster.stats.StatsSampleSpec
[JVM-Node1] Run starting. Expected test count is: 3
[JVM-Node1] StatsSampleSpecMultiJvmNode1:
[JVM-Node1] The stats sample
[JVM-Node3] Run starting. Expected test count is: 3
[JVM-Node2] Run starting. Expected test count is: 3
[JVM-Node3] StatsSampleSpecMultiJvmNode3:
[JVM-Node2] StatsSampleSpecMultiJvmNode2:
[JVM-Node3] The stats sample
[JVM-Node2] The stats sample
[JVM-Node1] - must illustrate how to startup cluster
[JVM-Node3] - must illustrate how to startup cluster
[JVM-Node2] - must illustrate how to startup cluster
[JVM-Node2] [WARN] [10/30/2012 06:15:18.011] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec@a5.local:60482/user/statsService] received dead letter from Actor[akka://StatsSampleSpec/system/testActor1]: StatsJob(this is the text that will be analyzed)
[JVM-Node2] [WARN] [10/30/2012 06:15:18.012] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec@a5.local:60482/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a5.local%3A60482]: Heartbeat(akka://StatsSampleSpec@a4.local:52502)
[JVM-Node2] - show usage of the statsService from one node *** FAILED ***
[JVM-Node2] java.lang.AssertionError: assertion failed: timeout (4979682013 nanoseconds) during expectMsg:
[JVM-Node2] at scala.Predef$.assert(Predef.scala:179)
[JVM-Node2] at akka.testkit.TestKitBase$class.expectMsgPF(TestKit.scala:312)
[JVM-Node2] at akka.testkit.TestKit.expectMsgPF(TestKit.scala:641)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4$$anonfun$apply$mcV$sp$5.apply$mcV$sp(StatsSampleSpec.scala:113)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec.runOn(MultiNodeSpec.scala:322)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4.apply$mcV$sp(StatsSampleSpec.scala:110)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4.apply(StatsSampleSpec.scala:109)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4.apply(StatsSampleSpec.scala:109)
[JVM-Node2] at akka.testkit.TestKitBase$class.within(TestKit.scala:262)
[JVM-Node2] at akka.testkit.TestKit.within(TestKit.scala:641)
[JVM-Node2] ...
[JVM-Node1] [ERROR] [10/30/2012 06:15:23.071] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec/user/controller/barriers] RoleName(second) tried to enter 'done-3' while we were waiting for 'done-2' (akka.remote.testconductor.BarrierCoordinator$WrongBarrier)
[JVM-Node2] - show usage of the statsService from all nodes *** FAILED ***
[JVM-Node2] java.lang.RuntimeException: barrier failed: done-3
[JVM-Node2] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node2] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node2] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node2] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node2] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node2] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node2] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node2] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node2] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node2] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node2] ...
[JVM-Node2] [WARN] [10/30/2012 06:15:23.082] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec/user/TestConductorClient] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: PoisonPill
[JVM-Node3] - show usage of the statsService from one node *** FAILED ***
[JVM-Node3] java.lang.RuntimeException: barrier failed: done-2
[JVM-Node3] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node3] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node3] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node3] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node3] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node3] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node3] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node3] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node3] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node3] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node3] ...
[JVM-Node2] [WARN] [10/30/2012 06:15:23.086] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec/user/statsWorker] received dead letter from Actor[akka://StatsSampleSpec@a5.local:60482/user/statsService/$b]: is
[JVM-Node1] - show usage of the statsService from one node *** FAILED ***
[JVM-Node1] java.lang.RuntimeException: barrier failed: done-2
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node1] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node1] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node1] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node1] ...
[JVM-Node2] Run completed in 7 seconds, 583 milliseconds.
[JVM-Node2] Total number of tests run: 3
[JVM-Node2] Suites: completed 1, aborted 0
[JVM-Node2] Tests: succeeded 1, failed 2, ignored 0, pending 0
[JVM-Node2] *** 2 TESTS FAILED ***
[JVM-Node1] - show usage of the statsService from all nodes *** FAILED ***
[JVM-Node1] java.lang.RuntimeException: barrier failed: done-3
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node1] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node1] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node1] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node1] ...
[JVM-Node1] [ERROR] [10/30/2012 06:15:23.190] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec/user/controller/barriers] cannot remove RoleName(first): no client to remove (akka.remote.testconductor.BarrierCoordinator$BarrierEmpty)
[JVM-Node1] [WARN] [10/30/2012 06:15:23.893] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec@a4.local:52502/system/cluster/core] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/coreSender]: GossipEnvelope(akka://StatsSampleSpec@a3.local:46272,Gossip(overview = GossipOverview(seen = [akka://StatsSampleSpec@a3.local:46272 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1), akka://StatsSampleSpec@a5.local:60482 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1), akka://StatsSampleSpec@a4.local:52502 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1)], unreachable = []), members = [Member(address = akka://StatsSampleSpec@a3.local:46272, status = Up), Member(address = akka://StatsSampleSpec@a4.local:52502, status = Up), Member(address = akka://StatsSampleSpec@a5.local:60482, status = Up)], version = VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1)),true)
[JVM-Node1] [WARN] [10/30/2012 06:15:23.894] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a3.local:46272)
[JVM-Node3] [WARN] [10/30/2012 06:15:23.966] [StatsSampleSpec-akka.actor.default-dispatcher-5] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a5.local:60482)
[JVM-Node1] [WARN] [10/30/2012 06:15:24.867] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:52502/system/cluster/core] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/coreSender]: GossipEnvelope(akka://StatsSampleSpec@a3.local:46272,Gossip(overview = GossipOverview(seen = [akka://StatsSampleSpec@a3.local:46272 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1), akka://StatsSampleSpec@a5.local:60482 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1), akka://StatsSampleSpec@a4.local:52502 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1)], unreachable = []), members = [Member(address = akka://StatsSampleSpec@a3.local:46272, status = Up), Member(address = akka://StatsSampleSpec@a4.local:52502, status = Up), Member(address = akka://StatsSampleSpec@a5.local:60482, status = Up)], version = VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1)),true)
[JVM-Node1] [WARN] [10/30/2012 06:15:24.901] [StatsSampleSpec-akka.actor.default-dispatcher-4] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a3.local:46272)
[JVM-Node3] [WARN] [10/30/2012 06:15:24.985] [StatsSampleSpec-akka.actor.default-dispatcher-4] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a5.local:60482)
[JVM-Node1] [WARN] [10/30/2012 06:15:25.890] [StatsSampleSpec-akka.actor.default-dispatcher-3] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a3.local:46272)
[JVM-Node3] [WARN] [10/30/2012 06:15:25.949] [StatsSampleSpec-akka.actor.default-dispatcher-4] [akka://StatsSampleSpec@a4.local:52502/system/cluster/metrics] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/metrics]: MetricsGossipEnvelope(akka://StatsSampleSpec@a5.local:60482,MetricsGossip(10,Set(NodeMetrics(akka://StatsSampleSpec@a3.local:46272,1351574125854,Set(Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,505217024,1351574119857,1351574125856))), Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,7497580544,1351574119857,1351574125855))), Metric(network-max-tx,None,None), Metric(processors,Some(16),None), Metric(total-cores,None,None), Metric(cpu-combined,None,None), Metric(system-load-average,Some(0.16),None), Metric(heap-memory-used,Some(65462584),Some(DataStream(10,62787904,1351574119857,1351574125856))), Metric(network-max-rx,None,None))), NodeMetrics(akka://StatsSampleSpec@a4.local:52502,1351574122941,Set(Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,1,1351574119952,1351574122944))), Metric(system-load-average,Some(0.43),None), Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,1,1351574119952,1351574122944))), Metric(network-max-tx,None,None), Metric(heap-memory-used,Some(54724624),Some(DataStream(10,674449,1351574119952,1351574122944))), Metric(processors,Some(16),None), Metric(total-cores,None,None), Metric(cpu-combined,None,None), Metric(network-max-rx,None,None))), NodeMetrics(akka://StatsSampleSpec@a5.local:60482,1351574125936,Set(Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,505217024,1351574119943,1351574125937))), Metric(network-max-tx,None,None), Metric(heap-memory-used,Some(63673296),Some(DataStream(10,60742576,1351574119943,1351574125937))), Metric(processors,Some(16),None), Metric(total-cores,None,None), Metric(cpu-combined,None,None), Metric(network-max-rx,None,None), Metric(system-load-average,Some(0.34),None), Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,7497580544,1351574119943,1351574125937))))))))
[JVM-Node3] [WARN] [10/30/2012 06:15:25.976] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a5.local:60482)
[JVM-Node1] [WARN] [10/30/2012 06:15:26.879] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a3.local:46272)
[JVM-Node3] [WARN] [10/30/2012 06:15:26.965] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a5.local:60482)
[JVM-Node1] [ERROR] [10/30/2012 06:15:27.865] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec/system/cluster/core] Cluster Node [akka://StatsSampleSpec@a3.local:46272] - Marking node(s) as UNREACHABLE [Member(address = akka://StatsSampleSpec@a4.local:52502, status = Up)]
[JVM-Node1] [WARN] [10/30/2012 06:15:27.904] [StatsSampleSpec-akka.actor.default-dispatcher-3] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: EndHeartbeat(akka://StatsSampleSpec@a3.local:46272)
[JVM-Node3] [WARN] [10/30/2012 06:15:27.989] [StatsSampleSpec-akka.actor.default-dispatcher-3] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: EndHeartbeat(akka://StatsSampleSpec@a5.local:60482)
[JVM-Node3] - show usage of the statsService from all nodes *** FAILED ***
[JVM-Node3] java.lang.AssertionError: assertion failed: timeout (4996789863 nanoseconds) during expectMsg:
[JVM-Node3] at scala.Predef$.assert(Predef.scala:179)
[JVM-Node3] at akka.testkit.TestKitBase$class.expectMsgPF(TestKit.scala:312)
[JVM-Node3] at akka.testkit.TestKit.expectMsgPF(TestKit.scala:641)
[JVM-Node3] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$6$$anonfun$apply$mcV$sp$7.apply$mcV$sp(StatsSampleSpec.scala:126)
[JVM-Node3] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$6$$anonfun$apply$mcV$sp$7.apply(StatsSampleSpec.scala:123)
[JVM-Node3] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$6$$anonfun$apply$mcV$sp$7.apply(StatsSampleSpec.scala:123)
[JVM-Node3] at akka.testkit.TestKitBase$class.within(TestKit.scala:262)
[JVM-Node3] at akka.testkit.TestKit.within(TestKit.scala:641)
[JVM-Node3] at akka.testkit.TestKitBase$class.within(TestKit.scala:276)
[JVM-Node3] at akka.testkit.TestKit.within(TestKit.scala:641)
[JVM-Node3] ...
[JVM-Node3] [WARN] [10/30/2012 06:15:28.088] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec/user/TestConductorClient] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: PoisonPill
[JVM-Node1] [WARN] [10/30/2012 06:15:28.115] [StatsSampleSpec-akka.actor.default-dispatcher-8] [akka://StatsSampleSpec/user/controller/10.0.1.233:35064-server1] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: ClientDisconnected
[JVM-Node1] [WARN] [10/30/2012 06:15:28.118] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec/user/TestConductorClient] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: PoisonPill
[JVM-Node1] [WARN] [10/30/2012 06:15:28.118] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec/user/controller] received dead letter from Actor[akka://StatsSampleSpec/user/controller/10.0.1.233:35064-server1]: ClientDisconnected(RoleName(first))
[JVM-Node3] Run completed in 12 seconds, 385 milliseconds.
[JVM-Node3] Total number of tests run: 3
[JVM-Node3] Suites: completed 1, aborted 0
[JVM-Node3] Tests: succeeded 1, failed 2, ignored 0, pending 0
[JVM-Node3] *** 2 TESTS FAILED ***
[JVM-Node1] Run completed in 12 seconds, 778 milliseconds.
[JVM-Node1] Total number of tests run: 3
[JVM-Node1] Suites: completed 1, aborted 0
[JVM-Node1] Tests: succeeded 1, failed 2, ignored 0, pending 0
[JVM-Node1] *** 2 TESTS FAILED ***
[error] Failed: sample.cluster.stats.StatsSampleSpecMultiJvmNode1
[error] Failed: sample.cluster.stats.StatsSampleSpecMultiJvmNode2
[error] Failed: sample.cluster.stats.StatsSampleSpecMultiJvmNode3
[info] * sample.cluster.stats.StatsSampleSpec
[JVM-Node1] Run starting. Expected test count is: 3
[JVM-Node1] StatsSampleSpecMultiJvmNode1:
[JVM-Node1] The stats sample
[JVM-Node3] Run starting. Expected test count is: 3
[JVM-Node2] Run starting. Expected test count is: 3
[JVM-Node3] StatsSampleSpecMultiJvmNode3:
[JVM-Node2] StatsSampleSpecMultiJvmNode2:
[JVM-Node3] The stats sample
[JVM-Node2] The stats sample
[JVM-Node1] - must illustrate how to startup cluster
[JVM-Node3] - must illustrate how to startup cluster
[JVM-Node2] - must illustrate how to startup cluster
[JVM-Node2] [WARN] [10/30/2012 06:15:18.011] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec@a5.local:60482/user/statsService] received dead letter from Actor[akka://StatsSampleSpec/system/testActor1]: StatsJob(this is the text that will be analyzed)
[JVM-Node2] [WARN] [10/30/2012 06:15:18.012] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec@a5.local:60482/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a5.local%3A60482]: Heartbeat(akka://StatsSampleSpec@a4.local:52502)
[JVM-Node2] - show usage of the statsService from one node *** FAILED ***
[JVM-Node2] java.lang.AssertionError: assertion failed: timeout (4979682013 nanoseconds) during expectMsg:
[JVM-Node2] at scala.Predef$.assert(Predef.scala:179)
[JVM-Node2] at akka.testkit.TestKitBase$class.expectMsgPF(TestKit.scala:312)
[JVM-Node2] at akka.testkit.TestKit.expectMsgPF(TestKit.scala:641)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4$$anonfun$apply$mcV$sp$5.apply$mcV$sp(StatsSampleSpec.scala:113)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec.runOn(MultiNodeSpec.scala:322)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4.apply$mcV$sp(StatsSampleSpec.scala:110)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4.apply(StatsSampleSpec.scala:109)
[JVM-Node2] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4.apply(StatsSampleSpec.scala:109)
[JVM-Node2] at akka.testkit.TestKitBase$class.within(TestKit.scala:262)
[JVM-Node2] at akka.testkit.TestKit.within(TestKit.scala:641)
[JVM-Node2] ...
[JVM-Node1] [ERROR] [10/30/2012 06:15:23.071] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec/user/controller/barriers] RoleName(second) tried to enter 'done-3' while we were waiting for 'done-2' (akka.remote.testconductor.BarrierCoordinator$WrongBarrier)
[JVM-Node2] - show usage of the statsService from all nodes *** FAILED ***
[JVM-Node2] java.lang.RuntimeException: barrier failed: done-3
[JVM-Node2] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node2] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node2] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node2] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node2] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node2] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node2] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node2] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node2] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node2] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node2] ...
[JVM-Node2] [WARN] [10/30/2012 06:15:23.082] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec/user/TestConductorClient] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: PoisonPill
[JVM-Node3] - show usage of the statsService from one node *** FAILED ***
[JVM-Node3] java.lang.RuntimeException: barrier failed: done-2
[JVM-Node3] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node3] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node3] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node3] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node3] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node3] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node3] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node3] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node3] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node3] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node3] ...
[JVM-Node2] [WARN] [10/30/2012 06:15:23.086] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec/user/statsWorker] received dead letter from Actor[akka://StatsSampleSpec@a5.local:60482/user/statsService/$b]: is
[JVM-Node1] - show usage of the statsService from one node *** FAILED ***
[JVM-Node1] java.lang.RuntimeException: barrier failed: done-2
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node1] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node1] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node1] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node1] ...
[JVM-Node2] Run completed in 7 seconds, 583 milliseconds.
[JVM-Node2] Total number of tests run: 3
[JVM-Node2] Suites: completed 1, aborted 0
[JVM-Node2] Tests: succeeded 1, failed 2, ignored 0, pending 0
[JVM-Node2] *** 2 TESTS FAILED ***
[JVM-Node1] - show usage of the statsService from all nodes *** FAILED ***
[JVM-Node1] java.lang.RuntimeException: barrier failed: done-3
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:201)
[JVM-Node1] at akka.remote.testconductor.ClientFSM$$anonfun$4.applyOrElse(Player.scala:177)
[JVM-Node1] at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
[JVM-Node1] at akka.actor.FSM$class.processEvent(FSM.scala:573)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.akka$actor$LoggingFSM$$super$processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.LoggingFSM$class.processEvent(FSM.scala:704)
[JVM-Node1] at akka.remote.testconductor.ClientFSM.processEvent(Player.scala:140)
[JVM-Node1] at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:567)
[JVM-Node1] at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
[JVM-Node1] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:408)
[JVM-Node1] ...
[JVM-Node1] [ERROR] [10/30/2012 06:15:23.190] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec/user/controller/barriers] cannot remove RoleName(first): no client to remove (akka.remote.testconductor.BarrierCoordinator$BarrierEmpty)
[JVM-Node1] [WARN] [10/30/2012 06:15:23.893] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec@a4.local:52502/system/cluster/core] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/coreSender]: GossipEnvelope(akka://StatsSampleSpec@a3.local:46272,Gossip(overview = GossipOverview(seen = [akka://StatsSampleSpec@a3.local:46272 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1), akka://StatsSampleSpec@a5.local:60482 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1), akka://StatsSampleSpec@a4.local:52502 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1)], unreachable = []), members = [Member(address = akka://StatsSampleSpec@a3.local:46272, status = Up), Member(address = akka://StatsSampleSpec@a4.local:52502, status = Up), Member(address = akka://StatsSampleSpec@a5.local:60482, status = Up)], version = VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1)),true)
[JVM-Node1] [WARN] [10/30/2012 06:15:23.894] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a3.local:46272)
[JVM-Node3] [WARN] [10/30/2012 06:15:23.966] [StatsSampleSpec-akka.actor.default-dispatcher-5] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a5.local:60482)
[JVM-Node1] [WARN] [10/30/2012 06:15:24.867] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:52502/system/cluster/core] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/coreSender]: GossipEnvelope(akka://StatsSampleSpec@a3.local:46272,Gossip(overview = GossipOverview(seen = [akka://StatsSampleSpec@a3.local:46272 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1), akka://StatsSampleSpec@a5.local:60482 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1), akka://StatsSampleSpec@a4.local:52502 -> VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1)], unreachable = []), members = [Member(address = akka://StatsSampleSpec@a3.local:46272, status = Up), Member(address = akka://StatsSampleSpec@a4.local:52502, status = Up), Member(address = akka://StatsSampleSpec@a5.local:60482, status = Up)], version = VectorClock(Node(4c89b07667d6f011f27b33cc773df015) -> 0000013ab0185df1)),true)
[JVM-Node1] [WARN] [10/30/2012 06:15:24.901] [StatsSampleSpec-akka.actor.default-dispatcher-4] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a3.local:46272)
[JVM-Node3] [WARN] [10/30/2012 06:15:24.985] [StatsSampleSpec-akka.actor.default-dispatcher-4] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a5.local:60482)
[JVM-Node1] [WARN] [10/30/2012 06:15:25.890] [StatsSampleSpec-akka.actor.default-dispatcher-3] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a3.local:46272)
[JVM-Node3] [WARN] [10/30/2012 06:15:25.949] [StatsSampleSpec-akka.actor.default-dispatcher-4] [akka://StatsSampleSpec@a4.local:52502/system/cluster/metrics] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/metrics]: MetricsGossipEnvelope(akka://StatsSampleSpec@a5.local:60482,MetricsGossip(10,Set(NodeMetrics(akka://StatsSampleSpec@a3.local:46272,1351574125854,Set(Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,505217024,1351574119857,1351574125856))), Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,7497580544,1351574119857,1351574125855))), Metric(network-max-tx,None,None), Metric(processors,Some(16),None), Metric(total-cores,None,None), Metric(cpu-combined,None,None), Metric(system-load-average,Some(0.16),None), Metric(heap-memory-used,Some(65462584),Some(DataStream(10,62787904,1351574119857,1351574125856))), Metric(network-max-rx,None,None))), NodeMetrics(akka://StatsSampleSpec@a4.local:52502,1351574122941,Set(Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,1,1351574119952,1351574122944))), Metric(system-load-average,Some(0.43),None), Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,1,1351574119952,1351574122944))), Metric(network-max-tx,None,None), Metric(heap-memory-used,Some(54724624),Some(DataStream(10,674449,1351574119952,1351574122944))), Metric(processors,Some(16),None), Metric(total-cores,None,None), Metric(cpu-combined,None,None), Metric(network-max-rx,None,None))), NodeMetrics(akka://StatsSampleSpec@a5.local:60482,1351574125936,Set(Metric(heap-memory-committed,Some(505217024),Some(DataStream(10,505217024,1351574119943,1351574125937))), Metric(network-max-tx,None,None), Metric(heap-memory-used,Some(63673296),Some(DataStream(10,60742576,1351574119943,1351574125937))), Metric(processors,Some(16),None), Metric(total-cores,None,None), Metric(cpu-combined,None,None), Metric(network-max-rx,None,None), Metric(system-load-average,Some(0.34),None), Metric(heap-memory-max,Some(7497580544),Some(DataStream(10,7497580544,1351574119943,1351574125937))))))))
[JVM-Node3] [WARN] [10/30/2012 06:15:25.976] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a5.local:60482)
[JVM-Node1] [WARN] [10/30/2012 06:15:26.879] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a3.local:46272)
[JVM-Node3] [WARN] [10/30/2012 06:15:26.965] [StatsSampleSpec-akka.actor.default-dispatcher-6] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: Heartbeat(akka://StatsSampleSpec@a5.local:60482)
[JVM-Node1] [ERROR] [10/30/2012 06:15:27.865] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec/system/cluster/core] Cluster Node [akka://StatsSampleSpec@a3.local:46272] - Marking node(s) as UNREACHABLE [Member(address = akka://StatsSampleSpec@a4.local:52502, status = Up)]
[JVM-Node1] [WARN] [10/30/2012 06:15:27.904] [StatsSampleSpec-akka.actor.default-dispatcher-3] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: EndHeartbeat(akka://StatsSampleSpec@a3.local:46272)
[JVM-Node3] [WARN] [10/30/2012 06:15:27.989] [StatsSampleSpec-akka.actor.default-dispatcher-3] [akka://StatsSampleSpec@a4.local:52502/system/cluster/heartbeatReceiver] received dead letter from Actor[akka://StatsSampleSpec/system/cluster/core/heartbeatSender/akka%3A%2F%2FStatsSampleSpec%40a4.local%3A52502]: EndHeartbeat(akka://StatsSampleSpec@a5.local:60482)
[JVM-Node3] - show usage of the statsService from all nodes *** FAILED ***
[JVM-Node3] java.lang.AssertionError: assertion failed: timeout (4996789863 nanoseconds) during expectMsg:
[JVM-Node3] at scala.Predef$.assert(Predef.scala:179)
[JVM-Node3] at akka.testkit.TestKitBase$class.expectMsgPF(TestKit.scala:312)
[JVM-Node3] at akka.testkit.TestKit.expectMsgPF(TestKit.scala:641)
[JVM-Node3] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$6$$anonfun$apply$mcV$sp$7.apply$mcV$sp(StatsSampleSpec.scala:126)
[JVM-Node3] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$6$$anonfun$apply$mcV$sp$7.apply(StatsSampleSpec.scala:123)
[JVM-Node3] at sample.cluster.stats.StatsSampleSpec$$anonfun$3$$anonfun$apply$mcV$sp$6$$anonfun$apply$mcV$sp$7.apply(StatsSampleSpec.scala:123)
[JVM-Node3] at akka.testkit.TestKitBase$class.within(TestKit.scala:262)
[JVM-Node3] at akka.testkit.TestKit.within(TestKit.scala:641)
[JVM-Node3] at akka.testkit.TestKitBase$class.within(TestKit.scala:276)
[JVM-Node3] at akka.testkit.TestKit.within(TestKit.scala:641)
[JVM-Node3] ...
[JVM-Node3] [WARN] [10/30/2012 06:15:28.088] [StatsSampleSpec-akka.actor.default-dispatcher-2] [akka://StatsSampleSpec/user/TestConductorClient] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: PoisonPill
[JVM-Node1] [WARN] [10/30/2012 06:15:28.115] [StatsSampleSpec-akka.actor.default-dispatcher-8] [akka://StatsSampleSpec/user/controller/10.0.1.233:35064-server1] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: ClientDisconnected
[JVM-Node1] [WARN] [10/30/2012 06:15:28.118] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec/user/TestConductorClient] received dead letter from Actor[akka://StatsSampleSpec/deadLetters]: PoisonPill
[JVM-Node1] [WARN] [10/30/2012 06:15:28.118] [StatsSampleSpec-akka.actor.default-dispatcher-1] [akka://StatsSampleSpec/user/controller] received dead letter from Actor[akka://StatsSampleSpec/user/controller/10.0.1.233:35064-server1]: ClientDisconnected(RoleName(first))
[JVM-Node3] Run completed in 12 seconds, 385 milliseconds.
[JVM-Node3] Total number of tests run: 3
[JVM-Node3] Suites: completed 1, aborted 0
[JVM-Node3] Tests: succeeded 1, failed 2, ignored 0, pending 0
[JVM-Node3] *** 2 TESTS FAILED ***
[JVM-Node1] Run completed in 12 seconds, 778 milliseconds.
[JVM-Node1] Total number of tests run: 3
[JVM-Node1] Suites: completed 1, aborted 0
[JVM-Node1] Tests: succeeded 1, failed 2, ignored 0, pending 0
[JVM-Node1] *** 2 TESTS FAILED ***
[error] Failed: sample.cluster.stats.StatsSampleSpecMultiJvmNode1
[error] Failed: sample.cluster.stats.StatsSampleSpecMultiJvmNode2
[error] Failed: sample.cluster.stats.StatsSampleSpecMultiJvmNode3
I looked at this, but I can't see how it could possibly be an error in the test. Unassigning me so that someone with fresh eyes can take a look.
The very interesting thing is the deadletters log
[akka://StatsSampleSpec@a5.local:60482/user/statsService] received dead letter from Actor[akka://StatsSampleSpec/system/testActor1]: StatsJob(this is the text that will be analyzed)
Doesn't that mean that when sending the message StatsJob to the remote node the lookup of the recipient fails (probably with Nobody)?
How can that happen?
Isn't an actor available to receive remote messages immediately after actorOf?
In code, isn't this ok?
system.actorOf(Props[StatsService], "statsService")
testConductor.enter("all-up")
val service = system.actorFor(node(third) / "user" / "statsService")
service ! StatsJob("this is the text that will be analyzed")
I might be looking at the wrong place, some fresh ideas would be great.
The very interesting thing is the deadletters log
[akka://StatsSampleSpec@a5.local:60482/user/statsService] received dead letter from Actor[akka://StatsSampleSpec/system/testActor1]: StatsJob(this is the text that will be analyzed)
Doesn't that mean that when sending the message StatsJob to the remote node the lookup of the recipient fails (probably with Nobody)?
How can that happen?
Isn't an actor available to receive remote messages immediately after actorOf?
In code, isn't this ok?
system.actorOf(Props[StatsService], "statsService")
testConductor.enter("all-up")
val service = system.actorFor(node(third) / "user" / "statsService")
service ! StatsJob("this is the text that will be analyzed")
I might be looking at the wrong place, some fresh ideas would be great.
on 2012-11-11 04:48 *
By viktorklang
Could theoretically be the same issue is in RoutingSpec
on 2012-11-16 08:04 *
By viktorklang
Here's fresher meat: https://jenkins.akka.io:8498/job/akka-release-2.1/33/consoleFull
on 2012-11-18 23:23 *
By bjorn.antonsson@typesafe.com
Assigned to set to bjorn.antonsson@typesafe.com
on 2012-11-19 00:03 *
By Patrik Nordwall
A few comments about the weekend tests. I added some logging to branch wip-2654-patriknw.
Jenkins jobs akka-local-repeat and akka-multi-node-repeat have run the tests 700+ times during the weekend. There are failing tests pretty often (a few % of the runs).
I have done some adjustments to the logging and the tests a few times. None of the changes are intended to be final, only debugging.
3 failure types remaining (after my adjustments).
- StatsSample
https://jenkins.akka.io:8498/job/akka-multi-node-repeat/331
- StatsSampleSingleMaster
https://jenkins.akka.io:8498/job/akka-multi-node-repeat/334
- TransformationSampleSpec
https://jenkins.akka.io:8498/job/akka-local-repeat/448
https://jenkins.akka.io:8498/job/akka-multi-node-repeat/385
Jenkins jobs akka-local-repeat and akka-multi-node-repeat have run the tests 700+ times during the weekend. There are failing tests pretty often (a few % of the runs).
I have done some adjustments to the logging and the tests a few times. None of the changes are intended to be final, only debugging.
3 failure types remaining (after my adjustments).
- StatsSample
https://jenkins.akka.io:8498/job/akka-multi-node-repeat/331
- StatsSampleSingleMaster
https://jenkins.akka.io:8498/job/akka-multi-node-repeat/334
- TransformationSampleSpec
https://jenkins.akka.io:8498/job/akka-local-repeat/448
https://jenkins.akka.io:8498/job/akka-multi-node-repeat/385
on 2012-11-20 01:32 *
By bjorn.antonsson@typesafe.com
So now we have one failure with debug logging enabled. Will attach files with failure and success for comparison.
on 2012-11-21 04:53 *
By bjorn.antonsson@typesafe.com
So the sequence of events that lead to the failures seems to be that two nodes try to connect to each other at almsot exactly the same time (Node-2 and Node-3 in Failure_630.txt).
What happens is that they both get active connections to the other one, and send CONNECT.
Node-2 gets the connect from Node-3 and changes his connection to a passive one (shutting down the active one)
Node-3 gets the connect from Node-2 and changes his connection to a passive one (shutting down the active one)
And then it just goes downhill from there.
What happens is that they both get active connections to the other one, and send CONNECT.
Node-2 gets the connect from Node-3 and changes his connection to a passive one (shutting down the active one)
Node-3 gets the connect from Node-2 and changes his connection to a passive one (shutting down the active one)
And then it just goes downhill from there.
The fixed version have been running for 200+ times each on multinode and multijvm versions on jenkins now with only one failure in multi-node (#2730) that doesn't seem to be related to the startup-race.
Will clean up the fixes in a separate branch.
Will clean up the fixes in a separate branch.
on 2012-11-22 03:01 *
By bjorn.antonsson@typesafe.com
Had to disable the two cluster tests that use throttling since it makes assumptions no longer happen to be true. Will be fixed in new remoting. See #2731.
on 2012-11-26 07:00 *
By bjorn.antonsson@typesafe.com
Milestone changed from Coltrane to 2.1
Status changed from Test to Fixed
