#2477
- Created on2012-09-06 12:37
- Reported by
- StatusFixed
- PriorityHigh (2)
- Assigned to
- Milestone
- Plan Level-
- Due Date-
Cluster error during creation of internal child actors - subscribe in preStart: Unsupported type of ActorRef for the recipient
Cluster Create problem in multi-jvm multi-node tests: I can only reproduce when running a test that runs clean 80% of the time, but sometimes I do see this:
[JVM-Node1] [INFO] [09/06/2012 06:23:30.608] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64519] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node1] [INFO] [09/06/2012 06:23:30.608] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64519] - has started up successfully
[JVM-Node2] [INFO] [09/06/2012 06:23:30.637] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - is starting up...
[JVM-Node2] [INFO] [09/06/2012 06:23:30.637] [main] [Cluster(akka://ClusterMetricsSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <= 'akka.cluster.scheduler.tick-duration' [33 ms].
[JVM-Node3] [INFO] [09/06/2012 06:23:30.690] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64520] - is starting up...
[JVM-Node3] [INFO] [09/06/2012 06:23:30.691] [main] [Cluster(akka://ClusterMetricsSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <= 'akka.cluster.scheduler.tick-duration' [33 ms].
[JVM-Node3] [INFO] [09/06/2012 06:23:30.871] [ClusterMetricsSpec-akka.actor.default-dispatcher-2] [akka://ClusterMetricsSpec/system/cluster/metrics] Metrics collection has started successfully on node [akka://ClusterMetricsSpec@localhost:64520]
[JVM-Node3] [INFO] [09/06/2012 06:23:30.885] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64520] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node3] [INFO] [09/06/2012 06:23:30.886] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64520] - has started up successfully
[JVM-Node2] [ERROR] [09/06/2012 06:23:30.914] [ClusterMetricsSpec-akka.actor.default-dispatcher-2] [akka://ClusterMetricsSpec/system/cluster/metrics] error while processing Create(-1029811383)
[JVM-Node2] 0b28a255-7c88-4835-a6ec-95064896edd5akka.actor.ActorInitializationException: exception during creation
[JVM-Node2] at akka.actor.ActorInitializationException$.apply(Actor.scala:145)
[JVM-Node2] at akka.actor.ActorCell.create(ActorCell.scala:454)
[JVM-Node2] at akka.actor.ActorCell.systemInvoke(ActorCell.scala:329)
[JVM-Node2] at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:256)
[JVM-Node2] at akka.dispatch.Mailbox.run(Mailbox.scala:211)
[JVM-Node2] at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
[JVM-Node2] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
[JVM-Node2] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
[JVM-Node2] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
[JVM-Node2] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
[JVM-Node2] Caused by: java.lang.IllegalArgumentException: Unsupported type of ActorRef for the recipient. Question not sent to [null]
[JVM-Node2] at akka.pattern.AskSupport$class.ask(AskSupport.scala:89)
[JVM-Node2] at akka.pattern.package$.ask(package.scala:41)
[JVM-Node2] at akka.pattern.AskSupport$AskableActorRef.$qmark(AskSupport.scala:155)
[JVM-Node2] at akka.cluster.Cluster.clusterCore$lzycompute(Cluster.scala:156)
[JVM-Node2] at akka.cluster.Cluster.clusterCore(Cluster.scala:152)
[JVM-Node2] at akka.cluster.Cluster.subscribe(Cluster.scala:190)
[JVM-Node2] at akka.cluster.ClusterNodeMetricsCollector.preStart(ClusterMetricsCollector.scala:106)
[JVM-Node2] at akka.actor.ActorCell.create(ActorCell.scala:444)
[JVM-Node2] ... 8 more
[JVM-Node2]
[JVM-Node2] [INFO] [09/06/2012 06:23:31.269] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node2] [INFO] [09/06/2012 06:23:31.270] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - has started up successfully
[JVM-Node2] [INFO] [09/06/2012 06:23:33.261] [ClusterMetricsSpec-akka.actor.default-dispatcher-3] [akka://ClusterMetricsSpec/system/cluster/core] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - Leader is moving node [akka://ClusterMetricsSpec@localhost:64516] from JOINING to UP
[JVM-Node2] [INFO] [09/06/2012 06:23:33.261] [ClusterMetricsSpec-akka.actor.default-dispatcher-3] [akka://ClusterMetricsSpec/system/cluster/core] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - Leader is moving node [akka://ClusterMetricsSpec@localhost:64519] from JOINING to UP
[JVM-Node2] [INFO] [09/06/2012 06:23:33.261] [ClusterMetricsSpec-akka.actor.default-dispatcher-3] [akka://ClusterMetricsSpec/system/cluster/core] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - Leader is moving node [akka://ClusterMetricsSpec@localhost:64520] from JOINING to UP
[JVM-Node1] [INFO] [09/06/2012 06:23:30.608] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64519] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node1] [INFO] [09/06/2012 06:23:30.608] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64519] - has started up successfully
[JVM-Node2] [INFO] [09/06/2012 06:23:30.637] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - is starting up...
[JVM-Node2] [INFO] [09/06/2012 06:23:30.637] [main] [Cluster(akka://ClusterMetricsSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <= 'akka.cluster.scheduler.tick-duration' [33 ms].
[JVM-Node3] [INFO] [09/06/2012 06:23:30.690] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64520] - is starting up...
[JVM-Node3] [INFO] [09/06/2012 06:23:30.691] [main] [Cluster(akka://ClusterMetricsSpec)] Using a dedicated scheduler for cluster. Default scheduler can be used if configured with 'akka.scheduler.tick-duration' [100 ms] <= 'akka.cluster.scheduler.tick-duration' [33 ms].
[JVM-Node3] [INFO] [09/06/2012 06:23:30.871] [ClusterMetricsSpec-akka.actor.default-dispatcher-2] [akka://ClusterMetricsSpec/system/cluster/metrics] Metrics collection has started successfully on node [akka://ClusterMetricsSpec@localhost:64520]
[JVM-Node3] [INFO] [09/06/2012 06:23:30.885] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64520] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node3] [INFO] [09/06/2012 06:23:30.886] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64520] - has started up successfully
[JVM-Node2] [ERROR] [09/06/2012 06:23:30.914] [ClusterMetricsSpec-akka.actor.default-dispatcher-2] [akka://ClusterMetricsSpec/system/cluster/metrics] error while processing Create(-1029811383)
[JVM-Node2] 0b28a255-7c88-4835-a6ec-95064896edd5akka.actor.ActorInitializationException: exception during creation
[JVM-Node2] at akka.actor.ActorInitializationException$.apply(Actor.scala:145)
[JVM-Node2] at akka.actor.ActorCell.create(ActorCell.scala:454)
[JVM-Node2] at akka.actor.ActorCell.systemInvoke(ActorCell.scala:329)
[JVM-Node2] at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:256)
[JVM-Node2] at akka.dispatch.Mailbox.run(Mailbox.scala:211)
[JVM-Node2] at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
[JVM-Node2] at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:262)
[JVM-Node2] at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
[JVM-Node2] at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
[JVM-Node2] at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
[JVM-Node2] Caused by: java.lang.IllegalArgumentException: Unsupported type of ActorRef for the recipient. Question not sent to [null]
[JVM-Node2] at akka.pattern.AskSupport$class.ask(AskSupport.scala:89)
[JVM-Node2] at akka.pattern.package$.ask(package.scala:41)
[JVM-Node2] at akka.pattern.AskSupport$AskableActorRef.$qmark(AskSupport.scala:155)
[JVM-Node2] at akka.cluster.Cluster.clusterCore$lzycompute(Cluster.scala:156)
[JVM-Node2] at akka.cluster.Cluster.clusterCore(Cluster.scala:152)
[JVM-Node2] at akka.cluster.Cluster.subscribe(Cluster.scala:190)
[JVM-Node2] at akka.cluster.ClusterNodeMetricsCollector.preStart(ClusterMetricsCollector.scala:106)
[JVM-Node2] at akka.actor.ActorCell.create(ActorCell.scala:444)
[JVM-Node2] ... 8 more
[JVM-Node2]
[JVM-Node2] [INFO] [09/06/2012 06:23:31.269] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - registered cluster JMX MBean [akka:type=Cluster]
[JVM-Node2] [INFO] [09/06/2012 06:23:31.270] [main] [Cluster(akka://ClusterMetricsSpec)] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - has started up successfully
[JVM-Node2] [INFO] [09/06/2012 06:23:33.261] [ClusterMetricsSpec-akka.actor.default-dispatcher-3] [akka://ClusterMetricsSpec/system/cluster/core] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - Leader is moving node [akka://ClusterMetricsSpec@localhost:64516] from JOINING to UP
[JVM-Node2] [INFO] [09/06/2012 06:23:33.261] [ClusterMetricsSpec-akka.actor.default-dispatcher-3] [akka://ClusterMetricsSpec/system/cluster/core] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - Leader is moving node [akka://ClusterMetricsSpec@localhost:64519] from JOINING to UP
[JVM-Node2] [INFO] [09/06/2012 06:23:33.261] [ClusterMetricsSpec-akka.actor.default-dispatcher-3] [akka://ClusterMetricsSpec/system/cluster/core] Cluster Node [akka://ClusterMetricsSpec@localhost:64516] - Leader is moving node [akka://ClusterMetricsSpec@localhost:64520] from JOINING to UP
Leave a comment
on 2012-09-07 07:21 *
By Patrik Nordwall
Assigned to set to Patrik Nordwall
Priority changed from Normal (3) to High (2)
on 2012-09-07 07:41 *
By Patrik Nordwall
The cause is that in AskSupport the ref is not an instance of InternalActorRef. This clusterDaemons ref is a bit special as it's a systemActorOf, which in the end creates a RepointableActorRef, but that is a InternalActorRef.
So I think clusterDaemons is null, and that could perhaps be possible since we had a leakage of 'this' before constructor is completed, i.e. threading visibility issue.
I think we should see if this doesn't go away when we merge in https://github.com/akka/akka/pull/675
So I think clusterDaemons is null, and that could perhaps be possible since we had a leakage of 'this' before constructor is completed, i.e. threading visibility issue.
I think we should see if this doesn't go away when we merge in https://github.com/akka/akka/pull/675
on 2012-09-18 12:16 *
By Helena Edelson
This did go away in the merge.
thanks for verifying that
Updating tickets (#939, #940, #1941, #2213, #2214, #2215, #2219, #2222, #2223, #2239, #2240, #2249, #2250, #2252, #2253, #2254, #2256, #2259, #2263, #2264, #2265, #2267, #2270, #2271, #2275, #2277, #2286, #2287, #2289, #2290, #2303, #2304, #2308, #2310, #2311, #2317, #2323, #2331, #2374, #2392, #2405, #2423, #2425, #2440, #2444, #2445, #2453, #2456, #2459, #2473, #2477, #2491, #2495, #2523, #2534, #2541, #2544, #2545, #2549, #2582, #2583, #2589, #2626)
Updating tickets (#939, #940, #1941, #2081, #2126, #2213, #2214, #2215, #2219, #2222, #2223, #2239, #2240, #2249, #2250, #2252, #2253, #2254, #2256, #2259, #2263, #2264, #2265, #2267, #2270, #2271, #2275, #2277, #2286, #2287, #2289, #2290, #2303, #2304, #2308, #2310, #2311, #2317, #2323, #2331, #2374, #2392, #2394, #2405, #2408, #2423, #2424, #2425, #2440, #2444, #2445, #2449, #2453, #2456, #2459, #2461, #2473, #2477, #2485, #2491, #2495, #2498, #2501, #2505, #2515, #2517, #2523, #2534, #2541, #2544, #2545, #2549, #2582, #2583, #2588, #2589, #2598, #2599, #2618, #2623, #2626, #2627, #2630, #2631, #2633, #2634, #2635, #2637, #2638, #2642, #2643, #2646, #2647, #2648, #2649, #2650, #2653, #2655, #2657, #2658)