#2143
  • Created on
    2012-05-28 13:11
  • Reported by
    Patrik Nordwall
  • Status
    Fixed
  • Priority
    Normal (3)
  • Assigned to
    rkuhn
  • Milestone
    2.1-RC1
  • Plan Level
    -
  • Due Date
    -

Attachments

Related Tickets

Add new subtask or bug

Initialization race in MultiNodeSpec

This failed when I run test in akka-remote-tests:
[info] * akka.remote.router.DirectRoutedRemoteActor
[JVM-Node1] DirectRoutedRemoteActorMultiJvmNode1:
[JVM-Node1] A new remote actor configured with a Direct router 
[JVM-Node1] [WARN] [05/28/2012 15:04:29.891] [MultiNodeSpec-akka.actor.default-dispatcher-4] [akka://MultiNodeSpec/user/TestConductorClient] received dead letter from Actor[akka://MultiNodeSpec/deadLetters]: PoisonPill
[JVM-Node2] [WARN] [05/28/2012 15:04:29.925] [MultiNodeSpec-akka.actor.default-dispatcher-5] [akka://MultiNodeSpec/user/TestConductorClient] received dead letter from Actor[akka://MultiNodeSpec/deadLetters]: ToServer(GetAddress(RoleName(slave)))
[JVM-Node2] *** RUN ABORTED ***
[JVM-Node2]   akka.pattern.AskTimeoutException: sending to terminated ref breaks promises
[JVM-Node2]   at akka.dispatch.DefaultPromise.result(Future.scala:861)
[JVM-Node2]   at akka.dispatch.Await$.result(Future.scala:75)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec$AwaitHelper.await(MultiNodeSpec.scala:192)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec.node(MultiNodeSpec.scala:185)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec$Replacement.addr(MultiNodeSpec.scala:209)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec$$anonfun$8$$anonfun$9.apply(MultiNodeSpec.scala:218)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec$$anonfun$8$$anonfun$9.apply(MultiNodeSpec.scala:214)
[JVM-Node2]   at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:143)
[JVM-Node2]   at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:143)
[JVM-Node2]   at scala.collection.Iterator$class.foreach(Iterator.scala:772)
[JVM-Node2]   at scala.collection.immutable.VectorIterator.foreach(Vector.scala:648)
[JVM-Node2]   at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
[JVM-Node2]   at scala.collection.immutable.Vector.foreach(Vector.scala:63)
[JVM-Node2]   at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:143)
[JVM-Node2]   at scala.collection.immutable.Vector.foldLeft(Vector.scala:63)
[JVM-Node2]   at scala.collection.TraversableOnce$class.$div$colon(TraversableOnce.scala:137)
[JVM-Node2]   at scala.collection.immutable.Vector.$div$colon(Vector.scala:63)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec$$anonfun$8.apply(MultiNodeSpec.scala:214)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec$$anonfun$8.apply(MultiNodeSpec.scala:213)
[JVM-Node2]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node2]   at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec.<init>(MultiNodeSpec.scala:213)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec.<init>(MultiNodeSpec.scala:137)
[JVM-Node2]   at akka.remote.router.DirectRoutedRemoteActorSpec.<init>(DirectRoutedRemoteActorMultiJvmSpec.scala:38)
[JVM-Node2]   at akka.remote.router.DirectRoutedRemoteActorMultiJvmNode2.<init>(DirectRoutedRemoteActorMultiJvmSpec.scala:36)
[JVM-Node2]   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[JVM-Node2]   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
[JVM-Node2]   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
[JVM-Node2]   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
[JVM-Node2]   at java.lang.Class.newInstance0(Class.java:355)
[JVM-Node2]   at java.lang.Class.newInstance(Class.java:308)
[JVM-Node2]   at org.scalatest.tools.Runner$$anonfun$23.apply(Runner.scala:1419)
[JVM-Node2]   at org.scalatest.tools.Runner$$anonfun$23.apply(Runner.scala:1416)
[JVM-Node2]   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
[JVM-Node2]   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
[JVM-Node2]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node2]   at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node2]   at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
[JVM-Node2]   at scala.collection.immutable.List.map(List.scala:76)
[JVM-Node2]   at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1416)
[JVM-Node2]   at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:584)
[JVM-Node2]   at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:583)
[JVM-Node2]   at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1558)
[JVM-Node2]   at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:582)
[JVM-Node2]   at org.scalatest.tools.Runner$.main(Runner.scala:466)
[JVM-Node2]   at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-Node2]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JVM-Node2]   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[JVM-Node2]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[JVM-Node2]   at java.lang.reflect.Method.invoke(Method.java:597)
[JVM-Node2]   at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
[JVM-Node2]   at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
[JVM-Node2]   at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
[JVM-Node2]   at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
[JVM-Node2]   at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
[JVM-Node2]   at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
[JVM-Node2]   at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
[JVM-Node2]   at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
[JVM-Node2]   at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
[JVM-Node2]   at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
[JVM-Node2]   at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[JVM-Node2]   Cause: akka.pattern.AskTimeoutException: sending to terminated ref breaks promises
[JVM-Node2]   at akka.pattern.AskSupport$class.ask(AskSupport.scala:76)
[JVM-Node2]   at akka.pattern.package$.ask(package.scala:43)
[JVM-Node2]   at akka.pattern.AskSupport$AskableActorRef.$qmark(AskSupport.scala:153)
[JVM-Node2]   at akka.remote.testconductor.Player$class.getAddressFor(Player.scala:92)
[JVM-Node2]   at akka.remote.testconductor.TestConductorExt.getAddressFor(Extension.scala:41)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec.node(MultiNodeSpec.scala:185)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec$Replacement.addr(MultiNodeSpec.scala:209)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec$$anonfun$8$$anonfun$9.apply(MultiNodeSpec.scala:218)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec$$anonfun$8$$anonfun$9.apply(MultiNodeSpec.scala:214)
[JVM-Node2]   at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:143)
[JVM-Node2]   at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:143)
[JVM-Node2]   at scala.collection.Iterator$class.foreach(Iterator.scala:772)
[JVM-Node2]   at scala.collection.immutable.VectorIterator.foreach(Vector.scala:648)
[JVM-Node2]   at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
[JVM-Node2]   at scala.collection.immutable.Vector.foreach(Vector.scala:63)
[JVM-Node2]   at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:143)
[JVM-Node2]   at scala.collection.immutable.Vector.foldLeft(Vector.scala:63)
[JVM-Node2]   at scala.collection.TraversableOnce$class.$div$colon(TraversableOnce.scala:137)
[JVM-Node2]   at scala.collection.immutable.Vector.$div$colon(Vector.scala:63)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec$$anonfun$8.apply(MultiNodeSpec.scala:214)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec$$anonfun$8.apply(MultiNodeSpec.scala:213)
[JVM-Node2]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node2]   at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec.<init>(MultiNodeSpec.scala:213)
[JVM-Node2]   at akka.remote.testkit.MultiNodeSpec.<init>(MultiNodeSpec.scala:137)
[JVM-Node2]   at akka.remote.router.DirectRoutedRemoteActorSpec.<init>(DirectRoutedRemoteActorMultiJvmSpec.scala:38)
[JVM-Node2]   at akka.remote.router.DirectRoutedRemoteActorMultiJvmNode2.<init>(DirectRoutedRemoteActorMultiJvmSpec.scala:36)
[JVM-Node2]   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[JVM-Node2]   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
[JVM-Node2]   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
[JVM-Node2]   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
[JVM-Node2]   at java.lang.Class.newInstance0(Class.java:355)
[JVM-Node2]   at java.lang.Class.newInstance(Class.java:308)
[JVM-Node2]   at org.scalatest.tools.Runner$$anonfun$23.apply(Runner.scala:1419)
[JVM-Node2]   at org.scalatest.tools.Runner$$anonfun$23.apply(Runner.scala:1416)
[JVM-Node2]   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
[JVM-Node2]   at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
[JVM-Node2]   at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
[JVM-Node2]   at scala.collection.immutable.List.foreach(List.scala:76)
[JVM-Node2]   at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
[JVM-Node2]   at scala.collection.immutable.List.map(List.scala:76)
[JVM-Node2]   at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1416)
[JVM-Node2]   at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:584)
[JVM-Node2]   at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:583)
[JVM-Node2]   at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1558)
[JVM-Node2]   at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:582)
[JVM-Node2]   at org.scalatest.tools.Runner$.main(Runner.scala:466)
[JVM-Node2]   at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-Node2]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JVM-Node2]   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[JVM-Node2]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[JVM-Node2]   at java.lang.reflect.Method.invoke(Method.java:597)
[JVM-Node2]   at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
[JVM-Node2]   at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
[JVM-Node2]   at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
[JVM-Node2]   at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
[JVM-Node2]   at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
[JVM-Node2]   at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
[JVM-Node2]   at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
[JVM-Node2]   at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
[JVM-Node2]   at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
[JVM-Node2]   at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
[JVM-Node2]   at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[error] Failed: akka.remote.router.DirectRoutedRemoteActorMultiJvmNode2
User picture
  on 2012-05-28 13:18 *
By Jonas Bonér
All these router tests should be rewritten to use the new multi-node tests and test conductor.
Create tickets for that and close this one as invalid.
User picture
  on 2012-05-28 13:27 *
By Patrik Nordwall
This test is already rewritten to multi-node. That is why it fails.

Now I understand better what is going on. It doesn't fail when running with ordinary sbt, but it fails when I run this with
sbt -Dakka.test.tags.include=long-running,timing

The test in itself is not tagged, and this means that all it's tests will be ignored, i.e. it's empty, and therefore test controller in node1 is probably exited before node2 is done with its init.

Hmm, I'm not sure how this should be handled.

Also, I think it is rather confusing that -Dakka.test.tags.include means that it will only run tests with the specified tag, but in some cases it might be useful to do that, of course.
User picture
  on 2012-05-28 13:34 *
By Jonas Bonér
I see. Thought it was the old one. Are all routing tests ported?

"Also, I think it is rather confusing that -Dakka.test.tags.include means that it will only run tests with the specified tag, but in some cases it might be useful to do that, of course."

I said this long time ago as well. Has to be fixed. Should run all normal plus the included tests.
User picture
  on 2012-05-28 13:36 *
By Patrik Nordwall
No, this is the only one ported so far. We have this ticket for porting the remote router tests: http://www.assembla.com/spaces/akka/tickets/1935

ok, I'll see what I can do about the tags.
User picture
  on 2012-05-29 09:43 *
By Patrik Nordwall
Assigned to set to Patrik Nordwall
(In revision:66efe504da5463c7292be6d364b89711a67652ae) Workaround for Initialization race in MultiNodeSpec, see #2143

Branch: wip-2143-MultiNodeSpec-init-race-patriknw
User picture
  on 2012-05-29 09:47 *
By Patrik Nordwall
Status changed from New to Test
I have created a workaround, https://github.com/akka/akka/pull/496
There might be a better way.
The problem is that when replacing deployment tags, which is done after startController/startClient, the node address is looked up, and if that happens after the controller node has exited the other nodes will fail with this exc.
In practice this is only a problem when we exclude/ignore all test cases - but that is something we need to be able to do.
User picture
  on 2012-05-29 10:51 *
By Patrik Nordwall
(In revision:7456bf595a374719c06dfe60b2a6825ee7cdaa50) Add doc note about using testconductor from constructor, see #2143

Branch: wip-2143-MultiNodeSpec-init-race-patriknw
User picture
  on 2012-05-29 12:17 *
By Patrik Nordwall
(In revision:66efe504da5463c7292be6d364b89711a67652ae) Workaround for Initialization race in MultiNodeSpec, see #2143

Branch: master
User picture
  on 2012-05-29 12:17 *
By Patrik Nordwall
(In revision:7456bf595a374719c06dfe60b2a6825ee7cdaa50) Add doc note about using testconductor from constructor, see #2143

Branch: master
User picture
  on 2012-05-29 12:17 *
By Patrik Nordwall
(In revision:eee96292e771c0a188d7c806cc57cce68a0c973d) Merge pull request #496 from akka/wip-2143-MultiNodeSpec-init-race-patriknw

Workaround for Initialization race in MultiNodeSpec, see #2143

Branch: master
User picture
  on 2012-05-29 12:34 *
By Patrik Nordwall
(In revision:66efe504da5463c7292be6d364b89711a67652ae) Workaround for Initialization race in MultiNodeSpec, see #2143

Branch: wip-2150-make-multi-jvm-tests-run-on-multiple-nodes
User picture
  on 2012-05-29 12:34 *
By Patrik Nordwall
(In revision:7456bf595a374719c06dfe60b2a6825ee7cdaa50) Add doc note about using testconductor from constructor, see #2143

Branch: wip-2150-make-multi-jvm-tests-run-on-multiple-nodes
User picture
  on 2012-05-29 12:34 *
By Patrik Nordwall
(In revision:eee96292e771c0a188d7c806cc57cce68a0c973d) Merge pull request #496 from akka/wip-2143-MultiNodeSpec-init-race-patriknw

Workaround for Initialization race in MultiNodeSpec, see #2143

Branch: wip-2150-make-multi-jvm-tests-run-on-multiple-nodes
User picture
  on 2012-05-29 12:37 *
By Patrik Nordwall
Assigned to changed from Patrik Nordwall to rkuhn
Roland, take a look at this when you have time later. I had to merge it to master, but feel free to revert and come up with a better solution. Would it be possible to ensure that controller node is not exited before other registered participants?
User picture
  on 2012-05-29 14:36 *
By Patrik Nordwall
(In revision:66efe504da5463c7292be6d364b89711a67652ae) Workaround for Initialization race in MultiNodeSpec, see #2143

Branch: wip-2134-deathwatch2.0-√
User picture
  on 2012-05-29 14:36 *
By Patrik Nordwall
(In revision:7456bf595a374719c06dfe60b2a6825ee7cdaa50) Add doc note about using testconductor from constructor, see #2143

Branch: wip-2134-deathwatch2.0-√
User picture
  on 2012-05-29 14:36 *
By Patrik Nordwall
(In revision:eee96292e771c0a188d7c806cc57cce68a0c973d) Merge pull request #496 from akka/wip-2143-MultiNodeSpec-init-race-patriknw

Workaround for Initialization race in MultiNodeSpec, see #2143

Branch: wip-2134-deathwatch2.0-√
User picture
  on 2012-06-14 12:53 *
By Jonas Bonér
Done?
User picture
  on 2012-07-05 13:53 *
By viktorklang
Milestone changed from Infrastructure and Framework For Distributed and Load Testing to Coltrane
Updating tickets (#2316, #641, #1936, #1939, #1941, #1942, #1945, #2010, #2049, #2052, #2058, #2120, #2121, #2126, #2140, #2142, #2143, #2149, #2151, #2176, #2274, #1792, #1940, #1943, #1112, #1938)
User picture
  on 2012-07-05 14:08 *
By Patrik Nordwall
Status changed from Test to Fixed
Closing this, it works.
User picture