LeaderDowningNodeThatIsUnreachableSpec - Race in cluster startup joining
LeaderDowningNodeThatIsUnreachableSpec failed 1 out of 13 runs on scalable1.
Git commit: 5578aef
Nodes are marked as unreachable directly in the beginning.
Note the error at the end
How/when can that happen?
Git commit: 5578aef
Nodes are marked as unreachable directly in the beginning.
Note the error at the end
cannot write GetAddress(RoleName(first)) while waiting for all-up
How/when can that happen?
[info] * akka.cluster.LeaderDowningNodeThatIsUnreachable
[JVM-Node1] [INFO] [06/01/2012 07:10:12.072] [main] [ActorSystem(MultiNodeSpec)] RemoteServerStarted@akka://MultiNodeSpec@localhost:53948
[JVM-Node2] [INFO] [06/01/2012 07:10:12.129] [main] [ActorSystem(MultiNodeSpec)] RemoteServerStarted@akka://MultiNodeSpec@localhost:47283
[JVM-Node3] [INFO] [06/01/2012 07:10:12.243] [main] [ActorSystem(MultiNodeSpec)] RemoteServerStarted@akka://MultiNodeSpec@localhost:55116
[JVM-Node4] [INFO] [06/01/2012 07:10:12.278] [main] [ActorSystem(MultiNodeSpec)] RemoteServerStarted@akka://MultiNodeSpec@localhost:56053
[JVM-Node1] LeaderDowningNodeThatIsUnreachableMultiJvmNode1:
[JVM-Node1] The Leader in a 4 node cluster
[JVM-Node1] [INFO] [06/01/2012 07:10:12.600] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:53948] - is starting up...
[JVM-Node3] LeaderDowningNodeThatIsUnreachableMultiJvmNode3:
[JVM-Node3] The Leader in a 4 node cluster
[JVM-Node4] LeaderDowningNodeThatIsUnreachableMultiJvmNode4:
[JVM-Node2] LeaderDowningNodeThatIsUnreachableMultiJvmNode2:
[JVM-Node2] The Leader in a 4 node cluster
[JVM-Node4] The Leader in a 4 node cluster
[JVM-Node1] [INFO] [06/01/2012 07:10:12.633] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:53948] - registering cluster JMX MBean [akka:type=Cluster]
[JVM-Node1] [INFO] [06/01/2012 07:10:12.634] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:53948] - has started up successfully
[JVM-Node3] [INFO] [06/01/2012 07:10:12.676] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:55116] - is starting up...
[JVM-Node2] [INFO] [06/01/2012 07:10:12.691] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - is starting up...
[JVM-Node4] [INFO] [06/01/2012 07:10:12.693] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:56053] - is starting up...
[JVM-Node3] [INFO] [06/01/2012 07:10:12.699] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:55116] - registering cluster JMX MBean [akka:type=Cluster]
[JVM-Node3] [INFO] [06/01/2012 07:10:12.699] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:55116] - has started up successfully
[JVM-Node3] [INFO] [06/01/2012 07:10:12.711] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:55116] - Trying to send JOIN to [akka://MultiNodeSpec@localhost:53948] through connection [Actor[akka://MultiNodeSpec@localhost:53948/system/cluster/commands]]
[JVM-Node4] [INFO] [06/01/2012 07:10:12.716] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:56053] - registering cluster JMX MBean [akka:type=Cluster]
[JVM-Node4] [INFO] [06/01/2012 07:10:12.717] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:56053] - has started up successfully
[JVM-Node2] [INFO] [06/01/2012 07:10:12.720] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - registering cluster JMX MBean [akka:type=Cluster]
[JVM-Node2] [INFO] [06/01/2012 07:10:12.720] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - has started up successfully
[JVM-Node4] [INFO] [06/01/2012 07:10:12.728] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:56053] - Trying to send JOIN to [akka://MultiNodeSpec@localhost:53948] through connection [Actor[akka://MultiNodeSpec@localhost:53948/system/cluster/commands]]
[JVM-Node2] [INFO] [06/01/2012 07:10:12.733] [main] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - Trying to send JOIN to [akka://MultiNodeSpec@localhost:53948] through connection [Actor[akka://MultiNodeSpec@localhost:53948/system/cluster/commands]]
[JVM-Node3] [INFO] [06/01/2012 07:10:12.754] [main] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:53948
[JVM-Node4] [INFO] [06/01/2012 07:10:12.774] [main] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:53948
[JVM-Node2] [INFO] [06/01/2012 07:10:12.777] [main] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:53948
[JVM-Node1] [INFO] [06/01/2012 07:10:12.804] [MultiNodeSpec-2] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:56053
[JVM-Node1] [INFO] [06/01/2012 07:10:12.806] [MultiNodeSpec-5] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:47283
[JVM-Node1] [INFO] [06/01/2012 07:10:12.806] [MultiNodeSpec-4] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:55116
[JVM-Node1] [INFO] [06/01/2012 07:10:12.835] [MultiNodeSpec-akka.actor.default-dispatcher-5] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:53948] - Node [akka://MultiNodeSpec@localhost:47283] is JOINING
[JVM-Node1] [INFO] [06/01/2012 07:10:12.839] [MultiNodeSpec-akka.actor.default-dispatcher-5] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:53948] - Node [akka://MultiNodeSpec@localhost:56053] is JOINING
[JVM-Node1] [INFO] [06/01/2012 07:10:12.839] [MultiNodeSpec-akka.actor.default-dispatcher-5] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:53948] - Node [akka://MultiNodeSpec@localhost:55116] is JOINING
[JVM-Node3] [INFO] [06/01/2012 07:10:13.337] [MultiNodeSpec-akka.actor.default-dispatcher-4] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:55116] - Leader is moving node [akka://MultiNodeSpec@localhost:55116] from JOINING to UP
[JVM-Node3] [INFO] [06/01/2012 07:10:13.339] [MultiNodeSpec-akka.actor.default-dispatcher-4] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:55116] - Leader is moving node [akka://MultiNodeSpec@localhost:55116] from JOINING to UP
[JVM-Node2] [INFO] [06/01/2012 07:10:13.344] [MultiNodeSpec-akka.actor.default-dispatcher-4] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:55116
[JVM-Node3] [INFO] [06/01/2012 07:10:13.355] [MultiNodeSpec-6] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:47283
[JVM-Node4] [INFO] [06/01/2012 07:10:13.380] [MultiNodeSpec-akka.actor.default-dispatcher-3] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:56053] - Leader is moving node [akka://MultiNodeSpec@localhost:56053] from JOINING to UP
[JVM-Node4] [INFO] [06/01/2012 07:10:13.381] [MultiNodeSpec-akka.actor.default-dispatcher-3] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:56053] - Leader is moving node [akka://MultiNodeSpec@localhost:56053] from JOINING to UP
[JVM-Node3] [INFO] [06/01/2012 07:10:13.939] [MultiNodeSpec-akka.actor.default-dispatcher-5] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:56053
[JVM-Node4] [INFO] [06/01/2012 07:10:13.946] [MultiNodeSpec-7] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:55116
[JVM-Node4] [INFO] [06/01/2012 07:10:15.182] [MultiNodeSpec-akka.actor.default-dispatcher-3] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:47283
[JVM-Node2] [INFO] [06/01/2012 07:10:15.191] [MultiNodeSpec-2] [ActorSystem(MultiNodeSpec)] RemoteClientStarted@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [INFO] [06/01/2012 07:10:16.040] [MultiNodeSpec-akka.actor.default-dispatcher-1] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - Marking node(s) as UNREACHABLE [Member(address = akka://MultiNodeSpec@localhost:53948, status = Joining)]
[JVM-Node2] [INFO] [06/01/2012 07:10:16.040] [MultiNodeSpec-akka.actor.default-dispatcher-3] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - Leader is moving node [akka://MultiNodeSpec@localhost:47283] from JOINING to UP
[JVM-Node2] [INFO] [06/01/2012 07:10:16.041] [MultiNodeSpec-akka.actor.default-dispatcher-3] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - Leader is moving node [akka://MultiNodeSpec@localhost:53948] from JOINING to UP
[JVM-Node2] [INFO] [06/01/2012 07:10:16.041] [MultiNodeSpec-akka.actor.default-dispatcher-3] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - Leader is moving node [akka://MultiNodeSpec@localhost:55116] from JOINING to UP
[JVM-Node2] [INFO] [06/01/2012 07:10:16.042] [MultiNodeSpec-akka.actor.default-dispatcher-3] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - Leader is marking unreachable node [akka://MultiNodeSpec@localhost:53948] as DOWN
[JVM-Node2] [INFO] [06/01/2012 07:10:17.240] [MultiNodeSpec-akka.actor.default-dispatcher-2] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - Leader is moving node [akka://MultiNodeSpec@localhost:47283] from JOINING to UP
[JVM-Node2] [INFO] [06/01/2012 07:10:17.240] [MultiNodeSpec-akka.actor.default-dispatcher-2] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - Leader is moving node [akka://MultiNodeSpec@localhost:55116] from JOINING to UP
[JVM-Node1] [INFO] [06/01/2012 07:10:22.583] [MultiNodeSpec-akka.actor.default-dispatcher-5] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:53948] - Marking node(s) as UNREACHABLE [Member(address = akka://MultiNodeSpec@localhost:55116, status = Joining)]
[JVM-Node2] [INFO] [06/01/2012 07:10:23.240] [MultiNodeSpec-akka.actor.default-dispatcher-2] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - Leader is marking unreachable node [akka://MultiNodeSpec@localhost:55116] as DOWN
[JVM-Node4] [INFO] [06/01/2012 07:10:26.281] [MultiNodeSpec-akka.actor.default-dispatcher-1] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:56053] - Marking node(s) as UNREACHABLE [Member(address = akka://MultiNodeSpec@localhost:55116, status = Up)]
[JVM-Node4] [INFO] [06/01/2012 07:10:28.980] [MultiNodeSpec-akka.actor.default-dispatcher-3] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:56053] - Marking node(s) as UNREACHABLE [Member(address = akka://MultiNodeSpec@localhost:53948, status = Joining)]
[JVM-Node1] - be able to DOWN a 'last' node that is UNREACHABLE *** FAILED ***
[JVM-Node1] java.lang.AssertionError: assertion failed: timeout 19796731038 nanoseconds expired
[JVM-Node1] at scala.Predef$.assert(Predef.scala:160)
[JVM-Node1] at akka.testkit.TestKit.poll$1(TestKit.scala:207)
[JVM-Node1] at akka.testkit.TestKit.awaitCond(TestKit.scala:213)
[JVM-Node1] at akka.cluster.MultiNodeClusterSpec$$anonfun$awaitUpConvergence$1.apply$mcV$sp(MultiNodeClusterSpec.scala:66)
[JVM-Node1] at akka.cluster.MultiNodeClusterSpec$$anonfun$awaitUpConvergence$1.apply(MultiNodeClusterSpec.scala:64)
[JVM-Node1] at akka.cluster.MultiNodeClusterSpec$$anonfun$awaitUpConvergence$1.apply(MultiNodeClusterSpec.scala:64)
[JVM-Node1] at akka.testkit.TestKit.within(TestKit.scala:244)
[JVM-Node1] at akka.testkit.TestKit.within(TestKit.scala:258)
[JVM-Node1] at akka.cluster.MultiNodeClusterSpec$class.awaitUpConvergence(MultiNodeClusterSpec.scala:64)
[JVM-Node1] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.awaitUpConvergence(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node1] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$2.apply$mcV$sp(LeaderDowningNodeThatIsUnreachableSpec.scala:48)
[JVM-Node1] at akka.remote.testkit.MultiNodeSpec.runOn(MultiNodeSpec.scala:179)
[JVM-Node1] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LeaderDowningNodeThatIsUnreachableSpec.scala:46)
[JVM-Node1] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(LeaderDowningNodeThatIsUnreachableSpec.scala:45)
[JVM-Node1] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(LeaderDowningNodeThatIsUnreachableSpec.scala:45)
[JVM-Node1] at org.scalatest.WordSpec$$anon$2.apply(WordSpec.scala:2161)
[JVM-Node1] at org.scalatest.Suite$class.withFixture(Suite.scala:1968)
[JVM-Node1] at akka.testkit.AkkaSpec.withFixture(AkkaSpec.scala:56)
[JVM-Node1] at org.scalatest.WordSpec$class.invokeWithFixture$1(WordSpec.scala:2158)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[JVM-Node1] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:168)
[JVM-Node1] at org.scalatest.WordSpec$class.runTest(WordSpec.scala:2167)
[JVM-Node1] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.org$scalatest$BeforeAndAfter$$super$runTest(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node1] at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:171)
[JVM-Node1] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.runTest(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[JVM-Node1] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:226)
[JVM-Node1] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[JVM-Node1] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node1] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node1] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[JVM-Node1] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:231)
[JVM-Node1] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[JVM-Node1] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node1] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node1] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[JVM-Node1] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:282)
[JVM-Node1] at org.scalatest.WordSpec$class.runTests(WordSpec.scala:2232)
[JVM-Node1] at akka.testkit.AkkaSpec.runTests(AkkaSpec.scala:56)
[JVM-Node1] at org.scalatest.Suite$class.run(Suite.scala:2286)
[JVM-Node1] at akka.testkit.AkkaSpec.org$scalatest$WordSpec$$super$run(AkkaSpec.scala:56)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[JVM-Node1] at org.scalatest.SuperEngine.runImpl(Engine.scala:318)
[JVM-Node1] at org.scalatest.WordSpec$class.run(WordSpec.scala:2279)
[JVM-Node1] at akka.testkit.AkkaSpec.org$scalatest$BeforeAndAfterAll$$super$run(AkkaSpec.scala:56)
[JVM-Node1] at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:213)
[JVM-Node1] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.org$scalatest$BeforeAndAfter$$super$run(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node1] at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:208)
[JVM-Node1] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.run(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node1] at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:59)
[JVM-Node1] at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1517)
[JVM-Node1] at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1514)
[JVM-Node1] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node1] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node1] at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1514)
[JVM-Node1] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:584)
[JVM-Node1] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:583)
[JVM-Node1] at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1558)
[JVM-Node1] at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:582)
[JVM-Node1] at org.scalatest.tools.Runner$.main(Runner.scala:466)
[JVM-Node1] at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-Node1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JVM-Node1] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[JVM-Node1] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[JVM-Node1] at java.lang.reflect.Method.invoke(Method.java:616)
[JVM-Node1] at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
[JVM-Node1] at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
[JVM-Node1] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
[JVM-Node1] at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
[JVM-Node1] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
[JVM-Node1] at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
[JVM-Node1] at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
[JVM-Node1] at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
[JVM-Node1] at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
[JVM-Node1] at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
[JVM-Node1] at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[JVM-Node3] - be able to DOWN a 'last' node that is UNREACHABLE *** FAILED ***
[JVM-Node3] akka.pattern.AskTimeoutException: Timed out
[JVM-Node3] at akka.dispatch.DefaultPromise.result(Future.scala:861)
[JVM-Node3] at akka.dispatch.Await$.result(Future.scala:75)
[JVM-Node3] at akka.remote.testconductor.Player$$anonfun$enter$1.apply(Player.scala:82)
[JVM-Node3] at akka.remote.testconductor.Player$$anonfun$enter$1.apply(Player.scala:80)
[JVM-Node3] at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
[JVM-Node3] at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
[JVM-Node3] at akka.remote.testconductor.Player$class.enter(Player.scala:80)
[JVM-Node3] at akka.remote.testconductor.TestConductorExt.enter(Extension.scala:41)
[JVM-Node3] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply$mcV$sp(LeaderDowningNodeThatIsUnreachableSpec.scala:76)
[JVM-Node3] at akka.remote.testkit.MultiNodeSpec.runOn(MultiNodeSpec.scala:179)
[JVM-Node3] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LeaderDowningNodeThatIsUnreachableSpec.scala:71)
[JVM-Node3] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(LeaderDowningNodeThatIsUnreachableSpec.scala:45)
[JVM-Node3] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(LeaderDowningNodeThatIsUnreachableSpec.scala:45)
[JVM-Node3] at org.scalatest.WordSpec$$anon$2.apply(WordSpec.scala:2161)
[JVM-Node3] at org.scalatest.Suite$class.withFixture(Suite.scala:1968)
[JVM-Node3] at akka.testkit.AkkaSpec.withFixture(AkkaSpec.scala:56)
[JVM-Node3] at org.scalatest.WordSpec$class.invokeWithFixture$1(WordSpec.scala:2158)
[JVM-Node3] at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[JVM-Node3] at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[JVM-Node3] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:168)
[JVM-Node3] at org.scalatest.WordSpec$class.runTest(WordSpec.scala:2167)
[JVM-Node3] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.org$scalatest$BeforeAndAfter$$super$runTest(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node3] at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:171)
[JVM-Node3] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.runTest(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node3] at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[JVM-Node3] at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[JVM-Node3] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:226)
[JVM-Node3] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[JVM-Node3] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node3] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node3] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[JVM-Node3] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:231)
[JVM-Node3] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[JVM-Node3] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node3] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node3] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[JVM-Node3] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:282)
[JVM-Node3] at org.scalatest.WordSpec$class.runTests(WordSpec.scala:2232)
[JVM-Node3] at akka.testkit.AkkaSpec.runTests(AkkaSpec.scala:56)
[JVM-Node3] at org.scalatest.Suite$class.run(Suite.scala:2286)
[JVM-Node3] at akka.testkit.AkkaSpec.org$scalatest$WordSpec$$super$run(AkkaSpec.scala:56)
[JVM-Node3] at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[JVM-Node3] at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[JVM-Node3] at org.scalatest.SuperEngine.runImpl(Engine.scala:318)
[JVM-Node3] at org.scalatest.WordSpec$class.run(WordSpec.scala:2279)
[JVM-Node3] at akka.testkit.AkkaSpec.org$scalatest$BeforeAndAfterAll$$super$run(AkkaSpec.scala:56)
[JVM-Node3] at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:213)
[JVM-Node3] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.org$scalatest$BeforeAndAfter$$super$run(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node3] at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:208)
[JVM-Node3] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.run(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node3] at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:59)
[JVM-Node3] at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1517)
[JVM-Node3] at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1514)
[JVM-Node3] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node3] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node3] at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1514)
[JVM-Node3] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:584)
[JVM-Node3] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:583)
[JVM-Node3] at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1558)
[JVM-Node3] at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:582)
[JVM-Node3] at org.scalatest.tools.Runner$.main(Runner.scala:466)
[JVM-Node3] at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-Node3] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JVM-Node3] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[JVM-Node3] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[JVM-Node3] at java.lang.reflect.Method.invoke(Method.java:616)
[JVM-Node3] at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
[JVM-Node3] at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
[JVM-Node3] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
[JVM-Node3] at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
[JVM-Node3] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
[JVM-Node3] at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
[JVM-Node3] at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
[JVM-Node3] at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
[JVM-Node3] at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
[JVM-Node3] at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
[JVM-Node3] at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[JVM-Node3] Cause: akka.pattern.AskTimeoutException: Timed out
[JVM-Node3] at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:277)
[JVM-Node3] at akka.actor.DefaultScheduler$$anon$4.run(Scheduler.scala:187)
[JVM-Node3] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:113)
[JVM-Node3] at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381)
[JVM-Node3] at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
[JVM-Node3] at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
[JVM-Node3] at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
[JVM-Node3] at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
[JVM-Node3] [ERROR] [06/01/2012 07:10:47.646] [MultiNodeSpec-akka.actor.default-dispatcher-5] [akka://MultiNodeSpec/user/TestConductorClient] cannot write GetAddress(RoleName(first)) while waiting for all-up
[JVM-Node1] [ERROR] [06/01/2012 07:10:47.684] [MultiNodeSpec-akka.actor.default-dispatcher-7] [akka://MultiNodeSpec/user/controller/barriers] timeout while waiting for barrier 'all-up' (akka.remote.testconductor.BarrierCoordinator$BarrierTimeout)
[JVM-Node4] - be able to DOWN a 'last' node that is UNREACHABLE *** FAILED ***
[JVM-Node4] akka.pattern.AskTimeoutException: Timed out
[JVM-Node4] at akka.dispatch.DefaultPromise.result(Future.scala:861)
[JVM-Node4] at akka.dispatch.Await$.result(Future.scala:75)
[JVM-Node4] at akka.remote.testconductor.Player$$anonfun$enter$1.apply(Player.scala:82)
[JVM-Node4] at akka.remote.testconductor.Player$$anonfun$enter$1.apply(Player.scala:80)
[JVM-Node4] at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
[JVM-Node4] at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
[JVM-Node4] at akka.remote.testconductor.Player$class.enter(Player.scala:80)
[JVM-Node4] at akka.remote.testconductor.TestConductorExt.enter(Extension.scala:41)
[JVM-Node4] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$3.apply$mcV$sp(LeaderDowningNodeThatIsUnreachableSpec.scala:68)
[JVM-Node4] at akka.remote.testkit.MultiNodeSpec.runOn(MultiNodeSpec.scala:179)
[JVM-Node4] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LeaderDowningNodeThatIsUnreachableSpec.scala:64)
[JVM-Node4] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(LeaderDowningNodeThatIsUnreachableSpec.scala:45)
[JVM-Node4] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(LeaderDowningNodeThatIsUnreachableSpec.scala:45)
[JVM-Node4] at org.scalatest.WordSpec$$anon$2.apply(WordSpec.scala:2161)
[JVM-Node4] at org.scalatest.Suite$class.withFixture(Suite.scala:1968)
[JVM-Node4] at akka.testkit.AkkaSpec.withFixture(AkkaSpec.scala:56)
[JVM-Node4] at org.scalatest.WordSpec$class.invokeWithFixture$1(WordSpec.scala:2158)
[JVM-Node4] at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[JVM-Node4] at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[JVM-Node4] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:168)
[JVM-Node4] at org.scalatest.WordSpec$class.runTest(WordSpec.scala:2167)
[JVM-Node4] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.org$scalatest$BeforeAndAfter$$super$runTest(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node4] at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:171)
[JVM-Node4] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.runTest(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node4] at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[JVM-Node4] at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[JVM-Node4] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:226)
[JVM-Node4] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[JVM-Node4] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node4] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node4] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[JVM-Node4] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:231)
[JVM-Node4] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[JVM-Node4] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node4] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node4] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[JVM-Node4] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:282)
[JVM-Node4] at org.scalatest.WordSpec$class.runTests(WordSpec.scala:2232)
[JVM-Node4] at akka.testkit.AkkaSpec.runTests(AkkaSpec.scala:56)
[JVM-Node4] at org.scalatest.Suite$class.run(Suite.scala:2286)
[JVM-Node4] at akka.testkit.AkkaSpec.org$scalatest$WordSpec$$super$run(AkkaSpec.scala:56)
[JVM-Node4] at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[JVM-Node4] at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[JVM-Node4] at org.scalatest.SuperEngine.runImpl(Engine.scala:318)
[JVM-Node4] at org.scalatest.WordSpec$class.run(WordSpec.scala:2279)
[JVM-Node4] at akka.testkit.AkkaSpec.org$scalatest$BeforeAndAfterAll$$super$run(AkkaSpec.scala:56)
[JVM-Node4] at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:213)
[JVM-Node4] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.org$scalatest$BeforeAndAfter$$super$run(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node4] at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:208)
[JVM-Node4] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.run(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node4] at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:59)
[JVM-Node4] at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1517)
[JVM-Node4] at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1514)
[JVM-Node4] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node4] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node4] at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1514)
[JVM-Node4] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:584)
[JVM-Node4] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:583)
[JVM-Node4] at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1558)
[JVM-Node4] at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:582)
[JVM-Node4] at org.scalatest.tools.Runner$.main(Runner.scala:466)
[JVM-Node4] at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-Node4] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JVM-Node4] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[JVM-Node4] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[JVM-Node4] at java.lang.reflect.Method.invoke(Method.java:616)
[JVM-Node4] at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
[JVM-Node4] at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
[JVM-Node4] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
[JVM-Node4] at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
[JVM-Node4] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
[JVM-Node4] at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
[JVM-Node4] at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
[JVM-Node4] at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
[JVM-Node4] at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
[JVM-Node4] at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
[JVM-Node4] at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[JVM-Node4] Cause: akka.pattern.AskTimeoutException: Timed out
[JVM-Node4] at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:277)
[JVM-Node4] at akka.actor.DefaultScheduler$$anon$4.run(Scheduler.scala:187)
[JVM-Node4] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:113)
[JVM-Node4] at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381)
[JVM-Node4] at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
[JVM-Node4] at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
[JVM-Node4] at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
[JVM-Node4] at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
[JVM-Node4] - must be able to DOWN a 'middle' node that is UNREACHABLE
[JVM-Node4] [WARN] [06/01/2012 07:10:48.008] [MultiNodeSpec-akka.actor.default-dispatcher-3] [akka://MultiNodeSpec/user/TestConductorClient] received dead letter from Actor[akka://MultiNodeSpec/deadLetters]: PoisonPill
[JVM-Node4] [INFO] [06/01/2012 07:10:48.026] [MultiNodeSpec-akka.actor.default-dispatcher-6] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:53948
[JVM-Node1] [INFO] [06/01/2012 07:10:48.033] [MultiNodeSpec-4] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] - be able to DOWN a 'last' node that is UNREACHABLE *** FAILED ***
[JVM-Node2] akka.pattern.AskTimeoutException: Timed out
[JVM-Node2] at akka.dispatch.DefaultPromise.result(Future.scala:861)
[JVM-Node2] at akka.dispatch.Await$.result(Future.scala:75)
[JVM-Node2] at akka.remote.testconductor.Player$$anonfun$enter$1.apply(Player.scala:82)
[JVM-Node2] at akka.remote.testconductor.Player$$anonfun$enter$1.apply(Player.scala:80)
[JVM-Node2] at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
[JVM-Node2] at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
[JVM-Node2] at akka.remote.testconductor.Player$class.enter(Player.scala:80)
[JVM-Node2] at akka.remote.testconductor.TestConductorExt.enter(Extension.scala:41)
[JVM-Node2] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply$mcV$sp(LeaderDowningNodeThatIsUnreachableSpec.scala:76)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec.runOn(MultiNodeSpec.scala:179)
[JVM-Node2] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LeaderDowningNodeThatIsUnreachableSpec.scala:71)
[JVM-Node2] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(LeaderDowningNodeThatIsUnreachableSpec.scala:45)
[JVM-Node2] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(LeaderDowningNodeThatIsUnreachableSpec.scala:45)
[JVM-Node2] at org.scalatest.WordSpec$$anon$2.apply(WordSpec.scala:2161)
[JVM-Node2] at org.scalatest.Suite$class.withFixture(Suite.scala:1968)
[JVM-Node2] at akka.testkit.AkkaSpec.withFixture(AkkaSpec.scala:56)
[JVM-Node2] at org.scalatest.WordSpec$class.invokeWithFixture$1(WordSpec.scala:2158)
[JVM-Node2] at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[JVM-Node2] at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2167)
[JVM-Node2] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:168)
[JVM-Node2] at org.scalatest.WordSpec$class.runTest(WordSpec.scala:2167)
[JVM-Node2] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.org$scalatest$BeforeAndAfter$$super$runTest(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node2] at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:171)
[JVM-Node2] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.runTest(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node2] at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[JVM-Node2] at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2232)
[JVM-Node2] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:226)
[JVM-Node2] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[JVM-Node2] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node2] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node2] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[JVM-Node2] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:231)
[JVM-Node2] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:215)
[JVM-Node2] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node2] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node2] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:215)
[JVM-Node2] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:282)
[JVM-Node2] at org.scalatest.WordSpec$class.runTests(WordSpec.scala:2232)
[JVM-Node2] at akka.testkit.AkkaSpec.runTests(AkkaSpec.scala:56)
[JVM-Node2] at org.scalatest.Suite$class.run(Suite.scala:2286)
[JVM-Node2] at akka.testkit.AkkaSpec.org$scalatest$WordSpec$$super$run(AkkaSpec.scala:56)
[JVM-Node2] at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[JVM-Node2] at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2279)
[JVM-Node2] at org.scalatest.SuperEngine.runImpl(Engine.scala:318)
[JVM-Node2] at org.scalatest.WordSpec$class.run(WordSpec.scala:2279)
[JVM-Node2] at akka.testkit.AkkaSpec.org$scalatest$BeforeAndAfterAll$$super$run(AkkaSpec.scala:56)
[JVM-Node2] at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:213)
[JVM-Node2] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.org$scalatest$BeforeAndAfter$$super$run(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node2] at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:208)
[JVM-Node2] at akka.cluster.LeaderDowningNodeThatIsUnreachableSpec.run(LeaderDowningNodeThatIsUnreachableSpec.scala:35)
[JVM-Node2] at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:59)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1517)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1514)
[JVM-Node2] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node2] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node2] at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1514)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:584)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:583)
[JVM-Node2] at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1558)
[JVM-Node2] at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:582)
[JVM-Node2] at org.scalatest.tools.Runner$.main(Runner.scala:466)
[JVM-Node2] at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-Node2] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JVM-Node2] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[JVM-Node2] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[JVM-Node2] at java.lang.reflect.Method.invoke(Method.java:616)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
[JVM-Node2] at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
[JVM-Node2] at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[JVM-Node2] Cause: akka.pattern.AskTimeoutException: Timed out
[JVM-Node2] at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:277)
[JVM-Node2] at akka.actor.DefaultScheduler$$anon$4.run(Scheduler.scala:187)
[JVM-Node2] at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:113)
[JVM-Node2] at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381)
[JVM-Node2] at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
[JVM-Node2] at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
[JVM-Node2] at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
[JVM-Node2] at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
[JVM-Node2] [ERROR] [06/01/2012 07:10:48.249] [MultiNodeSpec-akka.actor.default-dispatcher-2] [akka://MultiNodeSpec/user/TestConductorClient] cannot write GetAddress(RoleName(first)) while waiting for all-up
[JVM-Node2] [ERROR] [06/01/2012 07:10:48.367] [MultiNodeSpec-6] [ActorSystem(MultiNodeSpec)] RemoteServerError@akka://MultiNodeSpec@localhost:47283] Error[java.io.IOException: Connection reset by peer]
[JVM-Node2] [INFO] [06/01/2012 07:10:48.370] [MultiNodeSpec-6] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [ERROR] [06/01/2012 07:10:48.443] [MultiNodeSpec-akka.actor.default-dispatcher-4] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:48.444] [MultiNodeSpec-5] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:48.445] [MultiNodeSpec-akka.actor.default-dispatcher-1] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [ERROR] [06/01/2012 07:10:48.741] [MultiNodeSpec-akka.actor.default-dispatcher-2] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:48.742] [MultiNodeSpec-5] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:48.743] [MultiNodeSpec-akka.actor.default-dispatcher-4] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [ERROR] [06/01/2012 07:10:49.041] [MultiNodeSpec-akka.actor.default-dispatcher-4] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:49.042] [MultiNodeSpec-2] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:49.043] [MultiNodeSpec-akka.actor.default-dispatcher-1] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [ERROR] [06/01/2012 07:10:49.342] [MultiNodeSpec-akka.actor.default-dispatcher-4] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:49.342] [MultiNodeSpec-5] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:49.343] [MultiNodeSpec-akka.actor.default-dispatcher-1] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [ERROR] [06/01/2012 07:10:49.641] [MultiNodeSpec-akka.actor.default-dispatcher-1] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:49.642] [MultiNodeSpec-2] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:49.642] [MultiNodeSpec-akka.actor.default-dispatcher-2] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [ERROR] [06/01/2012 07:10:49.942] [MultiNodeSpec-akka.actor.default-dispatcher-1] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:49.942] [MultiNodeSpec-6] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:49.944] [MultiNodeSpec-akka.actor.default-dispatcher-2] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [ERROR] [06/01/2012 07:10:50.241] [MultiNodeSpec-akka.actor.default-dispatcher-2] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:50.242] [MultiNodeSpec-7] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:50.242] [MultiNodeSpec-akka.actor.default-dispatcher-4] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [ERROR] [06/01/2012 07:10:50.542] [MultiNodeSpec-akka.actor.default-dispatcher-4] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:50.542] [MultiNodeSpec-5] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:50.543] [MultiNodeSpec-akka.actor.default-dispatcher-1] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [ERROR] [06/01/2012 07:10:50.841] [MultiNodeSpec-akka.actor.default-dispatcher-4] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:50.849] [MultiNodeSpec-2] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:50.851] [MultiNodeSpec-akka.actor.default-dispatcher-5] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [ERROR] [06/01/2012 07:10:51.141] [MultiNodeSpec-akka.actor.default-dispatcher-4] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:51.141] [MultiNodeSpec-6] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:51.142] [MultiNodeSpec-akka.actor.default-dispatcher-5] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [ERROR] [06/01/2012 07:10:51.440] [MultiNodeSpec-akka.actor.default-dispatcher-4] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:51.441] [MultiNodeSpec-7] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:51.442] [MultiNodeSpec-akka.actor.default-dispatcher-5] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [INFO] [06/01/2012 07:10:51.739] [MultiNodeSpec-akka.actor.default-dispatcher-1] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - Marking node(s) as UNREACHABLE [Member(address = akka://MultiNodeSpec@localhost:56053, status = Up)]
[JVM-Node2] [ERROR] [06/01/2012 07:10:51.740] [MultiNodeSpec-akka.actor.default-dispatcher-5] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [ERROR] [06/01/2012 07:10:51.741] [MultiNodeSpec-5] [ActorSystem(MultiNodeSpec)] RemoteClientError@akka://MultiNodeSpec@localhost:56053: Error[java.net.ConnectException: Connection refused]
[JVM-Node2] [INFO] [06/01/2012 07:10:51.741] [MultiNodeSpec-akka.actor.default-dispatcher-1] [ActorSystem(MultiNodeSpec)] RemoteClientShutdown@akka://MultiNodeSpec@localhost:56053
[JVM-Node2] [INFO] [06/01/2012 07:10:52.040] [MultiNodeSpec-akka.actor.default-dispatcher-1] [Node(akka://MultiNodeSpec)] Cluster Node [akka://MultiNodeSpec@localhost:47283] - Leader is marking unreachable
Leave a comment
on 2012-06-01 09:08 *
By Jonas Bonér
You probably have to increase the threshold value in the FD.
This should also be a dilated value since it has to vary a lot between machines.
This should also be a dilated value since it has to vary a lot between machines.
on 2012-06-01 09:51 *
By Patrik Nordwall
Might be something we need to do, but here there is some "real remote errors", I would like to know what is causing them. First thing is to ask Roland about the
"cannot write GetAddress(RoleName(first)) while waiting for all-up"
"cannot write GetAddress(RoleName(first)) while waiting for all-up"
on 2012-06-01 09:57 *
By Jonas Bonér
Ok. But to investigate if the FD trips over to quick you can turn DEBUG on. The FD prints out its phi value for each heartbeat for each node so you can see if it reaches the threshold limit.
on 2012-06-01 10:48 *
By Patrik Nordwall
I changed to debug on this test.
on 2012-06-04 21:19 *
By Jonas Bonér
Summary changed from Race in cluster startup joining to LeaderDowningNodeThatIsUnreachableSpec - Race in cluster startup joining
on 2012-06-08 13:26 *
By bjorn.antonsson@typesafe.com
A quick comment on the "cannot write GetAddress(RoleName(first)) while waiting for all-up". It's the result of the clients ask timeout triggering before the barrier times out, and then the ClientFSM is still waiting for the barrier but the thread running the test is trying to do other things (like asking the test conductor for a nodes address).
Too old to spend time on.
Updating tickets (#620, #679, #725, #750, #752, #753, #754, #763, #789, #870, #893, #922, #953, #954, #971, #977, #983, #985, #987, #991, #1026, #1045, #1051, #1060, #1061, #1084, #1098, #1099, #1133, #1134, #1135, #1136, #1137, #1194, #1225, #1226, #1243, #1245, #1247, #1248, #1254, #1261, #1300, #1317, #1391, #1412, #1791, #1793, #1901, #1908, #1911, #1912, #1913, #1914, #1915, #1916, #1917, #1922, #1983, #1987, #1996, #1997, #1998, #2066, #2077, #2105, #2117, #2133, #2143, #2149, #2151, #2152, #2153, #2155, #2157, #2158, #2159, #2160, #2161, #2162, #2163, #2164, #2165, #2167, #2171, #2175, #2176, #2177, #2180, #2182, #2184, #2185, #2193, #2199, #2202, #2204, #2206, #2207, #2209, #2210)