Initialization race in MultiNodeSpec
This failed when I run test in akka-remote-tests:
[info] * akka.remote.router.DirectRoutedRemoteActor
[JVM-Node1] DirectRoutedRemoteActorMultiJvmNode1:
[JVM-Node1] A new remote actor configured with a Direct router
[JVM-Node1] [WARN] [05/28/2012 15:04:29.891] [MultiNodeSpec-akka.actor.default-dispatcher-4] [akka://MultiNodeSpec/user/TestConductorClient] received dead letter from Actor[akka://MultiNodeSpec/deadLetters]: PoisonPill
[JVM-Node2] [WARN] [05/28/2012 15:04:29.925] [MultiNodeSpec-akka.actor.default-dispatcher-5] [akka://MultiNodeSpec/user/TestConductorClient] received dead letter from Actor[akka://MultiNodeSpec/deadLetters]: ToServer(GetAddress(RoleName(slave)))
[JVM-Node2] *** RUN ABORTED ***
[JVM-Node2] akka.pattern.AskTimeoutException: sending to terminated ref breaks promises
[JVM-Node2] at akka.dispatch.DefaultPromise.result(Future.scala:861)
[JVM-Node2] at akka.dispatch.Await$.result(Future.scala:75)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec$AwaitHelper.await(MultiNodeSpec.scala:192)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec.node(MultiNodeSpec.scala:185)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec$Replacement.addr(MultiNodeSpec.scala:209)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec$$anonfun$8$$anonfun$9.apply(MultiNodeSpec.scala:218)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec$$anonfun$8$$anonfun$9.apply(MultiNodeSpec.scala:214)
[JVM-Node2] at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:143)
[JVM-Node2] at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:143)
[JVM-Node2] at scala.collection.Iterator$class.foreach(Iterator.scala:772)
[JVM-Node2] at scala.collection.immutable.VectorIterator.foreach(Vector.scala:648)
[JVM-Node2] at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
[JVM-Node2] at scala.collection.immutable.Vector.foreach(Vector.scala:63)
[JVM-Node2] at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:143)
[JVM-Node2] at scala.collection.immutable.Vector.foldLeft(Vector.scala:63)
[JVM-Node2] at scala.collection.TraversableOnce$class.$div$colon(TraversableOnce.scala:137)
[JVM-Node2] at scala.collection.immutable.Vector.$div$colon(Vector.scala:63)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec$$anonfun$8.apply(MultiNodeSpec.scala:214)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec$$anonfun$8.apply(MultiNodeSpec.scala:213)
[JVM-Node2] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node2] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec.<init>(MultiNodeSpec.scala:213)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec.<init>(MultiNodeSpec.scala:137)
[JVM-Node2] at akka.remote.router.DirectRoutedRemoteActorSpec.<init>(DirectRoutedRemoteActorMultiJvmSpec.scala:38)
[JVM-Node2] at akka.remote.router.DirectRoutedRemoteActorMultiJvmNode2.<init>(DirectRoutedRemoteActorMultiJvmSpec.scala:36)
[JVM-Node2] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[JVM-Node2] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
[JVM-Node2] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
[JVM-Node2] at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
[JVM-Node2] at java.lang.Class.newInstance0(Class.java:355)
[JVM-Node2] at java.lang.Class.newInstance(Class.java:308)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$23.apply(Runner.scala:1419)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$23.apply(Runner.scala:1416)
[JVM-Node2] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
[JVM-Node2] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
[JVM-Node2] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node2] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node2] at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
[JVM-Node2] at scala.collection.immutable.List.map(List.scala:76)
[JVM-Node2] at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1416)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:584)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:583)
[JVM-Node2] at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1558)
[JVM-Node2] at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:582)
[JVM-Node2] at org.scalatest.tools.Runner$.main(Runner.scala:466)
[JVM-Node2] at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-Node2] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JVM-Node2] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[JVM-Node2] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[JVM-Node2] at java.lang.reflect.Method.invoke(Method.java:597)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
[JVM-Node2] at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
[JVM-Node2] at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[JVM-Node2] Cause: akka.pattern.AskTimeoutException: sending to terminated ref breaks promises
[JVM-Node2] at akka.pattern.AskSupport$class.ask(AskSupport.scala:76)
[JVM-Node2] at akka.pattern.package$.ask(package.scala:43)
[JVM-Node2] at akka.pattern.AskSupport$AskableActorRef.$qmark(AskSupport.scala:153)
[JVM-Node2] at akka.remote.testconductor.Player$class.getAddressFor(Player.scala:92)
[JVM-Node2] at akka.remote.testconductor.TestConductorExt.getAddressFor(Extension.scala:41)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec.node(MultiNodeSpec.scala:185)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec$Replacement.addr(MultiNodeSpec.scala:209)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec$$anonfun$8$$anonfun$9.apply(MultiNodeSpec.scala:218)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec$$anonfun$8$$anonfun$9.apply(MultiNodeSpec.scala:214)
[JVM-Node2] at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:143)
[JVM-Node2] at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:143)
[JVM-Node2] at scala.collection.Iterator$class.foreach(Iterator.scala:772)
[JVM-Node2] at scala.collection.immutable.VectorIterator.foreach(Vector.scala:648)
[JVM-Node2] at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
[JVM-Node2] at scala.collection.immutable.Vector.foreach(Vector.scala:63)
[JVM-Node2] at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:143)
[JVM-Node2] at scala.collection.immutable.Vector.foldLeft(Vector.scala:63)
[JVM-Node2] at scala.collection.TraversableOnce$class.$div$colon(TraversableOnce.scala:137)
[JVM-Node2] at scala.collection.immutable.Vector.$div$colon(Vector.scala:63)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec$$anonfun$8.apply(MultiNodeSpec.scala:214)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec$$anonfun$8.apply(MultiNodeSpec.scala:213)
[JVM-Node2] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node2] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec.<init>(MultiNodeSpec.scala:213)
[JVM-Node2] at akka.remote.testkit.MultiNodeSpec.<init>(MultiNodeSpec.scala:137)
[JVM-Node2] at akka.remote.router.DirectRoutedRemoteActorSpec.<init>(DirectRoutedRemoteActorMultiJvmSpec.scala:38)
[JVM-Node2] at akka.remote.router.DirectRoutedRemoteActorMultiJvmNode2.<init>(DirectRoutedRemoteActorMultiJvmSpec.scala:36)
[JVM-Node2] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[JVM-Node2] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
[JVM-Node2] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
[JVM-Node2] at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
[JVM-Node2] at java.lang.Class.newInstance0(Class.java:355)
[JVM-Node2] at java.lang.Class.newInstance(Class.java:308)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$23.apply(Runner.scala:1419)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$23.apply(Runner.scala:1416)
[JVM-Node2] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
[JVM-Node2] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
[JVM-Node2] at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node2] at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node2] at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
[JVM-Node2] at scala.collection.immutable.List.map(List.scala:76)
[JVM-Node2] at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1416)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:584)
[JVM-Node2] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:583)
[JVM-Node2] at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1558)
[JVM-Node2] at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:582)
[JVM-Node2] at org.scalatest.tools.Runner$.main(Runner.scala:466)
[JVM-Node2] at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-Node2] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JVM-Node2] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[JVM-Node2] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[JVM-Node2] at java.lang.reflect.Method.invoke(Method.java:597)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
[JVM-Node2] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
[JVM-Node2] at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
[JVM-Node2] at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
[JVM-Node2] at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[error] Failed: akka.remote.router.DirectRoutedRemoteActorMultiJvmNode2
Leave a comment
on 2012-05-28 07:18 *
By Jonas Bonér
All these router tests should be rewritten to use the new multi-node tests and test conductor.
Create tickets for that and close this one as invalid.
Create tickets for that and close this one as invalid.
on 2012-05-28 07:27 *
By Patrik Nordwall
This test is already rewritten to multi-node. That is why it fails.
Now I understand better what is going on. It doesn't fail when running with ordinary sbt, but it fails when I run this with
sbt -Dakka.test.tags.include=long-running,timing
The test in itself is not tagged, and this means that all it's tests will be ignored, i.e. it's empty, and therefore test controller in node1 is probably exited before node2 is done with its init.
Hmm, I'm not sure how this should be handled.
Also, I think it is rather confusing that -Dakka.test.tags.include means that it will only run tests with the specified tag, but in some cases it might be useful to do that, of course.
Now I understand better what is going on. It doesn't fail when running with ordinary sbt, but it fails when I run this with
sbt -Dakka.test.tags.include=long-running,timing
The test in itself is not tagged, and this means that all it's tests will be ignored, i.e. it's empty, and therefore test controller in node1 is probably exited before node2 is done with its init.
Hmm, I'm not sure how this should be handled.
Also, I think it is rather confusing that -Dakka.test.tags.include means that it will only run tests with the specified tag, but in some cases it might be useful to do that, of course.
on 2012-05-28 07:34 *
By Jonas Bonér
I see. Thought it was the old one. Are all routing tests ported?
"Also, I think it is rather confusing that -Dakka.test.tags.include means that it will only run tests with the specified tag, but in some cases it might be useful to do that, of course."
I said this long time ago as well. Has to be fixed. Should run all normal plus the included tests.
"Also, I think it is rather confusing that -Dakka.test.tags.include means that it will only run tests with the specified tag, but in some cases it might be useful to do that, of course."
I said this long time ago as well. Has to be fixed. Should run all normal plus the included tests.
on 2012-05-28 07:36 *
By Patrik Nordwall
No, this is the only one ported so far. We have this ticket for porting the remote router tests: http://www.assembla.com/spaces/akka/tickets/1935
ok, I'll see what I can do about the tags.
ok, I'll see what I can do about the tags.
(In revision:66efe504da5463c7292be6d364b89711a67652ae) Workaround for Initialization race in MultiNodeSpec, see #2143
Branch: wip-2143-MultiNodeSpec-init-race-patriknw
Branch: wip-2143-MultiNodeSpec-init-race-patriknw
I have created a workaround, https://github.com/akka/akka/pull/496
There might be a better way.
The problem is that when replacing deployment tags, which is done after startController/startClient, the node address is looked up, and if that happens after the controller node has exited the other nodes will fail with this exc.
In practice this is only a problem when we exclude/ignore all test cases - but that is something we need to be able to do.
There might be a better way.
The problem is that when replacing deployment tags, which is done after startController/startClient, the node address is looked up, and if that happens after the controller node has exited the other nodes will fail with this exc.
In practice this is only a problem when we exclude/ignore all test cases - but that is something we need to be able to do.
on 2012-05-29 04:51 *
By Patrik Nordwall
(In revision:7456bf595a374719c06dfe60b2a6825ee7cdaa50) Add doc note about using testconductor from constructor, see #2143
Branch: wip-2143-MultiNodeSpec-init-race-patriknw
Branch: wip-2143-MultiNodeSpec-init-race-patriknw
on 2012-05-29 06:17 *
By Patrik Nordwall
(In revision:66efe504da5463c7292be6d364b89711a67652ae) Workaround for Initialization race in MultiNodeSpec, see #2143
Branch: master
Branch: master
on 2012-05-29 06:17 *
By Patrik Nordwall
(In revision:7456bf595a374719c06dfe60b2a6825ee7cdaa50) Add doc note about using testconductor from constructor, see #2143
Branch: master
Branch: master
on 2012-05-29 06:17 *
By Patrik Nordwall
(In revision:eee96292e771c0a188d7c806cc57cce68a0c973d) Merge pull request #496 from akka/wip-2143-MultiNodeSpec-init-race-patriknw
Workaround for Initialization race in MultiNodeSpec, see #2143
Branch: master
Workaround for Initialization race in MultiNodeSpec, see #2143
Branch: master
on 2012-05-29 06:34 *
By Patrik Nordwall
(In revision:66efe504da5463c7292be6d364b89711a67652ae) Workaround for Initialization race in MultiNodeSpec, see #2143
Branch: wip-2150-make-multi-jvm-tests-run-on-multiple-nodes
Branch: wip-2150-make-multi-jvm-tests-run-on-multiple-nodes
on 2012-05-29 06:34 *
By Patrik Nordwall
(In revision:7456bf595a374719c06dfe60b2a6825ee7cdaa50) Add doc note about using testconductor from constructor, see #2143
Branch: wip-2150-make-multi-jvm-tests-run-on-multiple-nodes
Branch: wip-2150-make-multi-jvm-tests-run-on-multiple-nodes
on 2012-05-29 06:34 *
By Patrik Nordwall
(In revision:eee96292e771c0a188d7c806cc57cce68a0c973d) Merge pull request #496 from akka/wip-2143-MultiNodeSpec-init-race-patriknw
Workaround for Initialization race in MultiNodeSpec, see #2143
Branch: wip-2150-make-multi-jvm-tests-run-on-multiple-nodes
Workaround for Initialization race in MultiNodeSpec, see #2143
Branch: wip-2150-make-multi-jvm-tests-run-on-multiple-nodes
Roland, take a look at this when you have time later. I had to merge it to master, but feel free to revert and come up with a better solution. Would it be possible to ensure that controller node is not exited before other registered participants?
on 2012-05-29 08:36 *
By Patrik Nordwall
(In revision:66efe504da5463c7292be6d364b89711a67652ae) Workaround for Initialization race in MultiNodeSpec, see #2143
Branch: wip-2134-deathwatch2.0-√
Branch: wip-2134-deathwatch2.0-√
on 2012-05-29 08:36 *
By Patrik Nordwall
(In revision:7456bf595a374719c06dfe60b2a6825ee7cdaa50) Add doc note about using testconductor from constructor, see #2143
Branch: wip-2134-deathwatch2.0-√
Branch: wip-2134-deathwatch2.0-√
on 2012-05-29 08:36 *
By Patrik Nordwall
(In revision:eee96292e771c0a188d7c806cc57cce68a0c973d) Merge pull request #496 from akka/wip-2143-MultiNodeSpec-init-race-patriknw
Workaround for Initialization race in MultiNodeSpec, see #2143
Branch: wip-2134-deathwatch2.0-√
Workaround for Initialization race in MultiNodeSpec, see #2143
Branch: wip-2134-deathwatch2.0-√
on 2012-06-14 06:53 *
By Jonas Bonér
Done?
on 2012-07-05 07:53 *
By viktorklang
Milestone changed from Infrastructure and Framework For Distributed and Load Testing to Coltrane
Closing this, it works.
Updating tickets (#620, #679, #725, #750, #752, #753, #754, #763, #789, #870, #893, #922, #953, #954, #971, #977, #983, #985, #987, #991, #1026, #1045, #1051, #1060, #1061, #1084, #1098, #1099, #1133, #1134, #1135, #1136, #1137, #1194, #1225, #1226, #1243, #1245, #1247, #1248, #1254, #1261, #1300, #1317, #1391, #1412, #1791, #1793, #1901, #1908, #1911, #1912, #1913, #1914, #1915, #1916, #1917, #1922, #1983, #1987, #1996, #1997, #1998, #2066, #2077, #2105, #2117, #2133, #2143, #2149, #2151, #2152, #2153, #2155, #2157, #2158, #2159, #2160, #2161, #2162, #2163, #2164, #2165, #2167, #2171, #2175, #2176, #2177, #2180, #2182, #2184, #2185, #2193, #2199, #2202, #2204, #2206, #2207, #2209, #2210)