#2175
- Created on2012-06-04 10:37
- Reported by
- StatusFixed
- PriorityHighest (1)
- Assigned to
- Milestone
- Plan Level-
- Due Date-
Problem with FaultInjector when blackholing communication
I have four nodes in the testconductor and tries to blackhole the communitcation between one of them and the other three nodes
The log is huge and I've cut and pasted stuff from it.
// These are the guys starting up
[JVM-Node1] RoleName(first): >>>> Normal: akka://MultiNodeSpec@localhost:56844
[JVM-Node2] RoleName(second): >>>> Normal: akka://MultiNodeSpec@localhost:56842
[JVM-Node3] RoleName(third): >>>> Normal: akka://MultiNodeSpec@localhost:56845
[JVM-Node4] RoleName(fourth): >>>> Victim: akka://MultiNodeSpec@localhost:56843
// After convergence we pull the plug by calling blackhole but only two out of three work ok
[JVM-Node1] RoleName(first): >>>>>>>>>> Pulling the plug from akka://MultiNodeSpec@localhost:56843 to akka://MultiNodeSpec@localhost:56844
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.733] [MultiNodeSpec-akka.actor.default-dispatcher-4] [akka://MultiNodeSpec/user/controller] received handled message Throttle(RoleName(fourth),RoleName(first),Both,0.0)
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.733] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/user/controller/127.0.0.1:56848-server3] processing Event(ToClient(ThrottleMsg(akka://MultiNodeSpec@localhost:56844,Both,0.0)),None) from Actor[akka://MultiNodeSpec/temp/$o]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.735] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] message from localhost/127.0.0.1:4711: ThrottleMsg(akka://MultiNodeSpec@localhost:56844,Both,0.0)
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.736] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ThrottleMsg(akka://MultiNodeSpec@localhost:56844,Both,0.0),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.744] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/system/FailureInjector/localhost:56844-snd1] processing Event(SetRate(0.0),Data(0,-1.0,Queue())) from Actor[akka://MultiNodeSpec/system/FailureInjector]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.744] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/system/FailureInjector/localhost:56844-snd1] transition PassThrough -> Blackhole
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.746] [MultiNodeSpec-akka.actor.default-dispatcher-5] [akka://MultiNodeSpec/system/FailureInjector/localhost:56844-rcv1] processing Event(SetRate(0.0),Data(0,-1.0,Queue())) from Actor[akka://MultiNodeSpec/system/FailureInjector]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.746] [MultiNodeSpec-akka.actor.default-dispatcher-5] [akka://MultiNodeSpec/system/FailureInjector/localhost:56844-rcv1] transition PassThrough -> Blackhole
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.746] [MultiNodeSpec-akka.actor.default-dispatcher-3] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ToServer(Done),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/user/TestConductorClient]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.747] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] channel [id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711] written 6
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.747] [New I/O server worker #2-3] [ConductorHandler(akka://MultiNodeSpec)] message from /127.0.0.1:56848: Done
[JVM-Node1] RoleName(first): >>>>>>>>>> Pulling the plug from akka://MultiNodeSpec@localhost:56843 to akka://MultiNodeSpec@localhost:56842
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.754] [MultiNodeSpec-akka.actor.default-dispatcher-1] [akka://MultiNodeSpec/user/controller] received handled message Throttle(RoleName(fourth),RoleName(second),Both,0.0)
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.754] [MultiNodeSpec-akka.actor.default-dispatcher-1] [akka://MultiNodeSpec/user/controller/127.0.0.1:56848-server3] processing Event(ToClient(ThrottleMsg(akka://MultiNodeSpec@localhost:56842,Both,0.0)),None) from Actor[akka://MultiNodeSpec/temp/$r]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.757] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] message from localhost/127.0.0.1:4711: ThrottleMsg(akka://MultiNodeSpec@localhost:56842,Both,0.0)
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.757] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ThrottleMsg(akka://MultiNodeSpec@localhost:56842,Both,0.0),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [MultiNodeSpec-akka.actor.default-dispatcher-2] [akka://MultiNodeSpec/system/FailureInjector/localhost:56842-snd3] processing Event(SetRate(0.0),Data(0,-1.0,Queue())) from Actor[akka://MultiNodeSpec/system/FailureInjector]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/system/FailureInjector/localhost:56842-rcv3] processing Event(SetRate(0.0),Data(0,-1.0,Queue())) from Actor[akka://MultiNodeSpec/system/FailureInjector]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [MultiNodeSpec-akka.actor.default-dispatcher-2] [akka://MultiNodeSpec/system/FailureInjector/localhost:56842-snd3] transition PassThrough -> Blackhole
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/system/FailureInjector/localhost:56842-rcv3] transition PassThrough -> Blackhole
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [MultiNodeSpec-akka.actor.default-dispatcher-3] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ToServer(Done),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/user/TestConductorClient]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] channel [id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711] written 6
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.759] [New I/O server worker #2-3] [ConductorHandler(akka://MultiNodeSpec)] message from /127.0.0.1:56848: Done
[JVM-Node1] RoleName(first): >>>>>>>>>> Pulling the plug from akka://MultiNodeSpec@localhost:56843 to akka://MultiNodeSpec@localhost:56845
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.768] [MultiNodeSpec-akka.actor.default-dispatcher-8] [akka://MultiNodeSpec/user/controller] received handled message Throttle(RoleName(fourth),RoleName(third),Both,0.0)
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.768] [MultiNodeSpec-akka.actor.default-dispatcher-8] [akka://MultiNodeSpec/user/controller/127.0.0.1:56848-server3] processing Event(ToClient(ThrottleMsg(akka://MultiNodeSpec@localhost:56845,Both,0.0)),None) from Actor[akka://MultiNodeSpec/temp/$u]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.770] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] message from localhost/127.0.0.1:4711: ThrottleMsg(akka://MultiNodeSpec@localhost:56845,Both,0.0)
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.771] [MultiNodeSpec-akka.actor.default-dispatcher-2] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ThrottleMsg(akka://MultiNodeSpec@localhost:56845,Both,0.0),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.774] [MultiNodeSpec-akka.actor.default-dispatcher-8] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ToServer(Done),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/user/TestConductorClient]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.774] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] channel [id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711] written 6
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.774] [New I/O server worker #2-3] [ConductorHandler(akka://MultiNodeSpec)] message from /127.0.0.1:56848: Done
The log is huge and I've cut and pasted stuff from it.
// These are the guys starting up
[JVM-Node1] RoleName(first): >>>> Normal: akka://MultiNodeSpec@localhost:56844
[JVM-Node2] RoleName(second): >>>> Normal: akka://MultiNodeSpec@localhost:56842
[JVM-Node3] RoleName(third): >>>> Normal: akka://MultiNodeSpec@localhost:56845
[JVM-Node4] RoleName(fourth): >>>> Victim: akka://MultiNodeSpec@localhost:56843
// After convergence we pull the plug by calling blackhole but only two out of three work ok
[JVM-Node1] RoleName(first): >>>>>>>>>> Pulling the plug from akka://MultiNodeSpec@localhost:56843 to akka://MultiNodeSpec@localhost:56844
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.733] [MultiNodeSpec-akka.actor.default-dispatcher-4] [akka://MultiNodeSpec/user/controller] received handled message Throttle(RoleName(fourth),RoleName(first),Both,0.0)
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.733] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/user/controller/127.0.0.1:56848-server3] processing Event(ToClient(ThrottleMsg(akka://MultiNodeSpec@localhost:56844,Both,0.0)),None) from Actor[akka://MultiNodeSpec/temp/$o]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.735] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] message from localhost/127.0.0.1:4711: ThrottleMsg(akka://MultiNodeSpec@localhost:56844,Both,0.0)
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.736] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ThrottleMsg(akka://MultiNodeSpec@localhost:56844,Both,0.0),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.744] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/system/FailureInjector/localhost:56844-snd1] processing Event(SetRate(0.0),Data(0,-1.0,Queue())) from Actor[akka://MultiNodeSpec/system/FailureInjector]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.744] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/system/FailureInjector/localhost:56844-snd1] transition PassThrough -> Blackhole
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.746] [MultiNodeSpec-akka.actor.default-dispatcher-5] [akka://MultiNodeSpec/system/FailureInjector/localhost:56844-rcv1] processing Event(SetRate(0.0),Data(0,-1.0,Queue())) from Actor[akka://MultiNodeSpec/system/FailureInjector]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.746] [MultiNodeSpec-akka.actor.default-dispatcher-5] [akka://MultiNodeSpec/system/FailureInjector/localhost:56844-rcv1] transition PassThrough -> Blackhole
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.746] [MultiNodeSpec-akka.actor.default-dispatcher-3] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ToServer(Done),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/user/TestConductorClient]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.747] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] channel [id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711] written 6
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.747] [New I/O server worker #2-3] [ConductorHandler(akka://MultiNodeSpec)] message from /127.0.0.1:56848: Done
[JVM-Node1] RoleName(first): >>>>>>>>>> Pulling the plug from akka://MultiNodeSpec@localhost:56843 to akka://MultiNodeSpec@localhost:56842
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.754] [MultiNodeSpec-akka.actor.default-dispatcher-1] [akka://MultiNodeSpec/user/controller] received handled message Throttle(RoleName(fourth),RoleName(second),Both,0.0)
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.754] [MultiNodeSpec-akka.actor.default-dispatcher-1] [akka://MultiNodeSpec/user/controller/127.0.0.1:56848-server3] processing Event(ToClient(ThrottleMsg(akka://MultiNodeSpec@localhost:56842,Both,0.0)),None) from Actor[akka://MultiNodeSpec/temp/$r]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.757] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] message from localhost/127.0.0.1:4711: ThrottleMsg(akka://MultiNodeSpec@localhost:56842,Both,0.0)
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.757] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ThrottleMsg(akka://MultiNodeSpec@localhost:56842,Both,0.0),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [MultiNodeSpec-akka.actor.default-dispatcher-2] [akka://MultiNodeSpec/system/FailureInjector/localhost:56842-snd3] processing Event(SetRate(0.0),Data(0,-1.0,Queue())) from Actor[akka://MultiNodeSpec/system/FailureInjector]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/system/FailureInjector/localhost:56842-rcv3] processing Event(SetRate(0.0),Data(0,-1.0,Queue())) from Actor[akka://MultiNodeSpec/system/FailureInjector]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [MultiNodeSpec-akka.actor.default-dispatcher-2] [akka://MultiNodeSpec/system/FailureInjector/localhost:56842-snd3] transition PassThrough -> Blackhole
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [MultiNodeSpec-akka.actor.default-dispatcher-6] [akka://MultiNodeSpec/system/FailureInjector/localhost:56842-rcv3] transition PassThrough -> Blackhole
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [MultiNodeSpec-akka.actor.default-dispatcher-3] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ToServer(Done),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/user/TestConductorClient]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.758] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] channel [id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711] written 6
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.759] [New I/O server worker #2-3] [ConductorHandler(akka://MultiNodeSpec)] message from /127.0.0.1:56848: Done
[JVM-Node1] RoleName(first): >>>>>>>>>> Pulling the plug from akka://MultiNodeSpec@localhost:56843 to akka://MultiNodeSpec@localhost:56845
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.768] [MultiNodeSpec-akka.actor.default-dispatcher-8] [akka://MultiNodeSpec/user/controller] received handled message Throttle(RoleName(fourth),RoleName(third),Both,0.0)
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.768] [MultiNodeSpec-akka.actor.default-dispatcher-8] [akka://MultiNodeSpec/user/controller/127.0.0.1:56848-server3] processing Event(ToClient(ThrottleMsg(akka://MultiNodeSpec@localhost:56845,Both,0.0)),None) from Actor[akka://MultiNodeSpec/temp/$u]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.770] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] message from localhost/127.0.0.1:4711: ThrottleMsg(akka://MultiNodeSpec@localhost:56845,Both,0.0)
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.771] [MultiNodeSpec-akka.actor.default-dispatcher-2] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ThrottleMsg(akka://MultiNodeSpec@localhost:56845,Both,0.0),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/deadLetters]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.774] [MultiNodeSpec-akka.actor.default-dispatcher-8] [akka://MultiNodeSpec/user/TestConductorClient] processing Event(ToServer(Done),Data(Some([id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711]),Some((unplug_victim_node,Actor[akka://MultiNodeSpec/temp/$n])))) from Actor[akka://MultiNodeSpec/user/TestConductorClient]
[JVM-Node4] [DEBUG] [06/04/2012 09:38:38.774] [New I/O client worker #2-1] [PlayerHandler(akka://MultiNodeSpec)] channel [id: 0x63f8247d, /127.0.0.1:56848 => localhost/127.0.0.1:4711] written 6
[JVM-Node1] [DEBUG] [06/04/2012 09:38:38.774] [New I/O server worker #2-3] [ConductorHandler(akka://MultiNodeSpec)] message from /127.0.0.1:56848: Done
Leave a comment
file:cfmgG6RIar4BW8acwqjQXA
The full log
The full log
on 2012-06-04 10:51 *
By bjorn.antonsson@typesafe.com
Assigned to set to bjorn.antonsson@typesafe.com
Status changed from New to Accepted
(In revision:9ee971ee794feff5944b880a21a54aa5ddec8948) We need to reregister a client conection when we know the actor system address. see #2175
Branch: wip-2175-problem-with-fault-injector-throttling-ban
Branch: wip-2175-problem-with-fault-injector-throttling-ban
So the problem was that when there is a client connection to an actor system, then we only know the client port during connection and not the real actor system address.
When we throttle the communication we use the actor system address, and consequently didn't find the connection to throttle.
Now we reregister unknown connections as soon as we know the real actor system address.
When we throttle the communication we use the actor system address, and consequently didn't find the connection to throttle.
Now we reregister unknown connections as soon as we know the real actor system address.
(In revision:9ee971ee794feff5944b880a21a54aa5ddec8948) We need to reregister a client conection when we know the actor system address. see #2175
Branch: master
Branch: master
Updating tickets (#620, #679, #725, #750, #752, #753, #754, #763, #789, #870, #893, #922, #953, #954, #971, #977, #983, #985, #987, #991, #1026, #1045, #1051, #1060, #1061, #1084, #1098, #1099, #1133, #1134, #1135, #1136, #1137, #1194, #1225, #1226, #1243, #1245, #1247, #1248, #1254, #1261, #1300, #1317, #1391, #1412, #1791, #1793, #1901, #1908, #1911, #1912, #1913, #1914, #1915, #1916, #1917, #1922, #1983, #1987, #1996, #1997, #1998, #2066, #2077, #2105, #2117, #2133, #2143, #2149, #2151, #2152, #2153, #2155, #2157, #2158, #2159, #2160, #2161, #2162, #2163, #2164, #2165, #2167, #2171, #2175, #2176, #2177, #2180, #2182, #2184, #2185, #2193, #2199, #2202, #2204, #2206, #2207, #2209, #2210)