SingletonClusterWithFailureDetectorPuppet failure on Scalable1
http://217.142.157.108:8080/job/akka-local/338
[info] * akka.cluster.SingletonClusterWithFailureDetectorPuppet
[JVM-Node1] [INFO] [10/01/2012 21:50:57.188] [main] [SingletonClusterWithFailureDetectorPuppetMultiJvmNode1(akka://SingletonClusterSpec)] Role [first] started with address [akka://SingletonClusterSpec@localhost:58816]
[JVM-Node2] [INFO] [10/01/2012 21:50:57.272] [main] [SingletonClusterWithFailureDetectorPuppetMultiJvmNode2(akka://SingletonClusterSpec)] Role [second] started with address [akka://SingletonClusterSpec@localhost:33510]
[JVM-Node1] SingletonClusterWithFailureDetectorPuppetMultiJvmNode1:
[JVM-Node1] A cluster of 2 nodes
[JVM-Node1] [INFO] [10/01/2012 21:50:57.304] [main] [Cluster(akka://SingletonClusterSpec)] Cluster Node [akka://SingletonClusterSpec@localhost:58816] - is starting up...
[JVM-Node1] [INFO] [10/01/2012 21:50:57.306] [main] [Cluster(akka://SingletonClusterSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <= 'akka.cluster.scheduler.tick-duration' [33 ms].
[JVM-Node2] SingletonClusterWithFailureDetectorPuppetMultiJvmNode2:
[JVM-Node1] [INFO] [10/01/2012 21:50:57.361] [main] [Cluster(akka://SingletonClusterSpec)] Cluster Node [akka://SingletonClusterSpec@localhost:58816] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node1] [INFO] [10/01/2012 21:50:57.362] [main] [Cluster(akka://SingletonClusterSpec)] Cluster Node [akka://SingletonClusterSpec@localhost:58816] - has started up successfully
[JVM-Node2] A cluster of 2 nodes
[JVM-Node1] [INFO] [10/01/2012 21:50:57.368] [SingletonClusterSpec-akka.actor.default-dispatcher-2] [akka://SingletonClusterSpec/system/cluster/metrics] Hyperic SIGAR was not found on the classpath or not installed properly. Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriateplatform-specific native libary to 'java.library.path'.
[JVM-Node1] [INFO] [10/01/2012 21:50:57.378] [SingletonClusterSpec-akka.actor.default-dispatcher-2] [akka://SingletonClusterSpec/system/cluster/metrics] Metrics collection has started successfully on node [akka://SingletonClusterSpec@localhost:58816]
[JVM-Node2] [INFO] [10/01/2012 21:50:57.380] [main] [Cluster(akka://SingletonClusterSpec)] Cluster Node [akka://SingletonClusterSpec@localhost:33510] - is starting up...
[JVM-Node2] [INFO] [10/01/2012 21:50:57.382] [main] [Cluster(akka://SingletonClusterSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <= 'akka.cluster.scheduler.tick-duration' [33 ms].
[JVM-Node1] [WARN] [10/01/2012 21:50:57.403] [SingletonClusterSpec-akka.actor.default-dispatcher-1] [akka://SingletonClusterSpec/system/cluster/core] unhandled message from Actor[akka://SingletonClusterSpec/system/cluster/core]: JoinSeedNodes(Vector())
[JVM-Node2] [INFO] [10/01/2012 21:50:57.436] [main] [Cluster(akka://SingletonClusterSpec)] Cluster Node [akka://SingletonClusterSpec@localhost:33510] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node2] [INFO] [10/01/2012 21:50:57.436] [main] [Cluster(akka://SingletonClusterSpec)] Cluster Node [akka://SingletonClusterSpec@localhost:33510] - has started up successfully
[JVM-Node2] [INFO] [10/01/2012 21:50:57.447] [SingletonClusterSpec-akka.actor.default-dispatcher-2] [akka://SingletonClusterSpec/system/cluster/metrics] Hyperic SIGAR was not found on the classpath or not installed properly. Metrics will be retreived from MBeans, and may be incorrect on some platforms. To increase metric accuracy add the 'sigar.jar' to the classpath and the appropriateplatform-specific native libary to 'java.library.path'.
[JVM-Node2] [INFO] [10/01/2012 21:50:57.456] [SingletonClusterSpec-akka.actor.default-dispatcher-2] [akka://SingletonClusterSpec/system/cluster/metrics] Metrics collection has started successfully on node [akka://SingletonClusterSpec@localhost:33510]
[JVM-Node2] [WARN] [10/01/2012 21:50:57.481] [SingletonClusterSpec-akka.actor.default-dispatcher-1] [akka://SingletonClusterSpec/system/cluster/core] unhandled message from Actor[akka://SingletonClusterSpec/system/cluster/core]: JoinSeedNodes(Vector())
[JVM-Node1] [INFO] [10/01/2012 21:50:57.712] [SingletonClusterSpec-akka.actor.default-dispatcher-1] [akka://SingletonClusterSpec/system/cluster/core] Cluster Node [akka://SingletonClusterSpec@localhost:58816] - Leader is moving node [akka://SingletonClusterSpec@localhost:58816] from JOINING to UP
[JVM-Node2] [INFO] [10/01/2012 21:50:57.791] [SingletonClusterSpec-akka.actor.default-dispatcher-5] [akka://SingletonClusterSpec/system/cluster/core] Cluster Node [akka://SingletonClusterSpec@localhost:33510] - Leader is moving node [akka://SingletonClusterSpec@localhost:33510] from JOINING to UP
[JVM-Node2] - must become singleton cluster when started with 'auto-join=on' and 'seed-nodes=[]' (504 milliseconds)
[JVM-Node1] - must become singleton cluster when started with 'auto-join=on' and 'seed-nodes=[]' (581 milliseconds)
[JVM-Node2] [INFO] [10/01/2012 21:50:58.185] [SingletonClusterSpec-akka.actor.default-dispatcher-4] [akka://SingletonClusterSpec/system/cluster/core] Cluster Node [akka://SingletonClusterSpec@localhost:33510] - Leader is moving node [akka://SingletonClusterSpec@localhost:33510] from JOINING to UP
[JVM-Node1] - must not be singleton cluster when joined with other node (528 milliseconds)
[JVM-Node2] - must not be singleton cluster when joined with other node (528 milliseconds)
[JVM-Node1] [ERROR] [10/01/2012 21:50:58.502] [SingletonClusterSpec-akka.actor.default-dispatcher-6] [akka://SingletonClusterSpec/system/cluster/core] Cluster Node [akka://SingletonClusterSpec@localhost:58816] - Marking node(s) as UNREACHABLE [Member(address = akka://SingletonClusterSpec@localhost:33510, status = Up)]
[JVM-Node1] - become singleton cluster when one node is shutdown *** FAILED *** (109 milliseconds)
[JVM-Node1] false was not true (MultiNodeClusterSpec.scala:167)
[JVM-Node1] org.scalatest.exceptions.TestFailedException:
[JVM-Node1] at org.scalatest.matchers.Matchers$class.newTestFailedException(Matchers.scala:155)
[JVM-Node1] at akka.cluster.SingletonClusterSpec.newTestFailedException(SingletonClusterSpec.scala:34)
[JVM-Node1] at org.scalatest.matchers.MustMatchers$MustMethodHelper$.mustMatcher(MustMatchers.scala:884)
[JVM-Node1] at org.scalatest.matchers.MustMatchers$AnyMustWrapper.must(MustMatchers.scala:912)
[JVM-Node1] at akka.cluster.MultiNodeClusterSpec$class.assertLeaderIn(MultiNodeClusterSpec.scala:167)
[JVM-Node1] at akka.cluster.SingletonClusterSpec.assertLeaderIn(SingletonClusterSpec.scala:34)
[JVM-Node1] at akka.cluster.MultiNodeClusterSpec$class.assertLeader(MultiNodeClusterSpec.scala:156)
[JVM-Node1] at akka.cluster.SingletonClusterSpec.assertLeader(SingletonClusterSpec.scala:34)
[JVM-Node1] at akka.cluster.SingletonClusterSpec$$anonfun$1$$anonfun$apply$mcV$sp$3$$anonfun$apply$mcV$sp$4.apply$mcV$sp(SingletonClusterSpec.scala:69)
[JVM-Node1] at akka.remote.testkit.MultiNodeSpec.runOn(MultiNodeSpec.scala:302)
[JVM-Node1] at akka.cluster.SingletonClusterSpec$$anonfun$1$$anonfun$apply$mcV$sp$3.apply$mcV$sp(SingletonClusterSpec.scala:61)
[JVM-Node1] at akka.cluster.SingletonClusterSpec$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(SingletonClusterSpec.scala:60)
[JVM-Node1] at akka.cluster.SingletonClusterSpec$$anonfun$1$$anonfun$apply$mcV$sp$3.apply(SingletonClusterSpec.scala:60)
[JVM-Node1] at org.scalatest.WordSpec$$anon$2.apply(WordSpec.scala:2179)
[JVM-Node1] at org.scalatest.Suite$class.withFixture(Suite.scala:1974)
[JVM-Node1] at akka.cluster.SingletonClusterSpec.akka$cluster$MultiNodeClusterSpec$$super$withFixture(SingletonClusterSpec.scala:34)
[JVM-Node1] at akka.cluster.MultiNodeClusterSpec$class.withFixture(MultiNodeClusterSpec.scala:87)
[JVM-Node1] at akka.cluster.SingletonClusterSpec.withFixture(SingletonClusterSpec.scala:34)
[JVM-Node1] at org.scalatest.WordSpec$class.invokeWithFixture$1(WordSpec.scala:2176)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2185)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$runTest$1.apply(WordSpec.scala:2185)
[JVM-Node1] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:198)
[JVM-Node1] at org.scalatest.WordSpec$class.runTest(WordSpec.scala:2185)
[JVM-Node1] at akka.cluster.SingletonClusterSpec.runTest(SingletonClusterSpec.scala:34)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2250)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$runTests$1.apply(WordSpec.scala:2250)
[JVM-Node1] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:260)
[JVM-Node1] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:249)
[JVM-Node1] at scala.collection.immutable.List.foreach(List.scala:309)
[JVM-Node1] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:249)
[JVM-Node1] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:265)
[JVM-Node1] at org.scalatest.SuperEngine$$anonfun$org$scalatest$SuperEngine$$runTestsInBranch$1.apply(Engine.scala:249)
[JVM-Node1] at scala.collection.immutable.List.foreach(List.scala:309)
[JVM-Node1] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:249)
[JVM-Node1] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:326)
[JVM-Node1] at org.scalatest.WordSpec$class.runTests(WordSpec.scala:2250)
[JVM-Node1] at akka.cluster.SingletonClusterSpec.runTests(SingletonClusterSpec.scala:34)
[JVM-Node1] at org.scalatest.Suite$class.run(Suite.scala:2303)
[JVM-Node1] at akka.cluster.SingletonClusterSpec.org$scalatest$WordSpec$$super$run(SingletonClusterSpec.scala:34)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2297)
[JVM-Node1] at org.scalatest.WordSpec$$anonfun$run$1.apply(WordSpec.scala:2297)
[JVM-Node1] at org.scalatest.SuperEngine.runImpl(Engine.scala:362)
[JVM-Node1] at org.scalatest.WordSpec$class.run(WordSpec.scala:2297)
[JVM-Node1] at akka.cluster.SingletonClusterSpec.org$scalatest$BeforeAndAfterAll$$super$run(SingletonClusterSpec.scala:34)
[JVM-Node1] at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:213)
[JVM-Node1] at akka.cluster.SingletonClusterSpec.run(SingletonClusterSpec.scala:34)
[JVM-Node1] at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:60)
[JVM-Node1] at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1560)
[JVM-Node1] at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:1557)
[JVM-Node1] at scala.collection.immutable.List.foreach(List.scala:309)
[JVM-Node1] at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1557)
[JVM-Node1] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:697)
[JVM-Node1] at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:696)
[JVM-Node1] at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1601)
[JVM-Node1] at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:695)
[JVM-Node1] at org.scalatest.tools.Runner$.main(Runner.scala:571)
[JVM-Node1] at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-Node1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JVM-Node1] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[JVM-Node1] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[JVM-Node1] at java.lang.reflect.Method.invoke(Method.java:616)
[JVM-Node1] at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:71)
[JVM-Node1] at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
[JVM-Node1] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:139)
[JVM-Node1] at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:71)
[JVM-Node1] at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:139)
[JVM-Node1] at scala.tools.nsc.CommonRunner$class.run(ObjectRunner.scala:28)
[JVM-Node1] at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:45)
[JVM-Node1] at scala.tools.nsc.CommonRunner$class.runAndCatch(ObjectRunner.scala:35)
[JVM-Node1] at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:45)
[JVM-Node1] at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:74)
[JVM-Node1] at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:96)
[JVM-Node1] at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:105)
[JVM-Node1] at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
[JVM-Node1] [INFO] [10/01/2012 21:50:58.699] [SingletonClusterSpec-akka.actor.default-dispatcher-6] [akka://SingletonClusterSpec/system/cluster/core] Cluster Node [akka://SingletonClusterSpec@localhost:58816] - Leader is marking unreachable node [akka://SingletonClusterSpec@localhost:33510] as DOWN
[JVM-Node1] [WARN] [10/01/2012 21:50:58.819] [SingletonClusterSpec-akka.actor.default-dispatcher-5] [akka://SingletonClusterSpec/user/controller/127.0.0.1:35832-server1] received dead letter from Actor[akka://SingletonClusterSpec/deadLetters]: ClientDisconnected
[JVM-Node1] [WARN] [10/01/2012 21:50:58.820] [SingletonClusterSpec-akka.actor.default-dispatcher-5] [akka://SingletonClusterSpec/user/controller] received dead letter from Actor[akka://SingletonClusterSpec/user/controller/127.0.0.1:35832-server1]: ClientDisconnected(RoleName(first))
[JVM-Node1] [WARN] [10/01/2012 21:50:58.822] [SingletonClusterSpec-akka.actor.default-dispatcher-5] [akka://SingletonClusterSpec/user/TestConductorClient] received dead letter from Actor[akka://SingletonClusterSpec/deadLetters]: PoisonPill
[JVM-Node1] [INFO] [10/01/2012 21:50:58.845] [SingletonClusterSpec-akka.actor.default-dispatcher-4] [Cluster(akka://SingletonClusterSpec)] Cluster Node [akka://SingletonClusterSpec@localhost:58816] - Shutting down cluster Node and cluster daemons...
[JVM-Node1] [INFO] [10/01/2012 21:50:58.850] [SingletonClusterSpec-akka.actor.default-dispatcher-4] [Cluster(akka://SingletonClusterSpec)] Cluster Node [akka://SingletonClusterSpec@localhost:58816] - Cluster node successfully shut down
[error] Failed: akka.cluster.SingletonClusterWithFailureDetectorPuppetMultiJvmNode1
Leave a comment
This is tricky. I have added some debug logging and started testing on scalable1 to gather more information.
on 2012-10-02 14:07 *
By Patrik Nordwall
Component changed from None to cluster-tests
Milestone changed from 2.1-RC1 to Coltrane
Changing milestone, since it's not critical for 2.1 release
on 2012-10-03 07:46 *
By Patrik Nordwall
Was run 50 times without failure yesterday evening. Will continue to run today and hopefully have a failure with the extra debug logging.
on 2012-10-04 05:28 *
By Patrik Nordwall
No failures after 200 runs. The extra logging probably hides the race.
Updating tickets (#939, #940, #1941, #2213, #2214, #2215, #2219, #2222, #2223, #2239, #2240, #2249, #2250, #2252, #2253, #2254, #2256, #2259, #2263, #2264, #2265, #2267, #2270, #2271, #2275, #2277, #2286, #2287, #2289, #2290, #2303, #2304, #2308, #2310, #2311, #2317, #2323, #2331, #2374, #2392, #2405, #2423, #2425, #2440, #2444, #2445, #2453, #2456, #2459, #2473, #2477, #2491, #2495, #2523, #2534, #2541, #2544, #2545, #2549, #2582, #2583, #2589, #2626)
Updating tickets (#939, #940, #1941, #2081, #2126, #2213, #2214, #2215, #2219, #2222, #2223, #2239, #2240, #2249, #2250, #2252, #2253, #2254, #2256, #2259, #2263, #2264, #2265, #2267, #2270, #2271, #2275, #2277, #2286, #2287, #2289, #2290, #2303, #2304, #2308, #2310, #2311, #2317, #2323, #2331, #2374, #2392, #2394, #2405, #2408, #2423, #2424, #2425, #2440, #2444, #2445, #2449, #2453, #2456, #2459, #2461, #2473, #2477, #2485, #2491, #2495, #2498, #2501, #2505, #2515, #2517, #2523, #2534, #2541, #2544, #2545, #2549, #2582, #2583, #2588, #2589, #2598, #2599, #2618, #2623, #2626, #2627, #2630, #2631, #2633, #2634, #2635, #2637, #2638, #2642, #2643, #2646, #2647, #2648, #2649, #2650, #2653, #2655, #2657, #2658)