This is from a mailing list thread that I started...

We have encountered what seems like it might be a bit of a bottleneck in akka-remote:

akka:event-driven:dispatcher:global-12 [BLOCKED]
akka.remote.netty.RemoteClient.send(Object, Option, Option, InetSocketAddress, long, boolean, ActorRef, Option, ActorType)
akka.remote.netty.NettyRemoteClientModule$$anonfun$send$1.apply(RemoteClient)
akka.remote.netty.NettyRemoteClientModule$$anonfun$send$1.apply(Object)
akka.remote.netty.NettyRemoteClientModule$class.withClientFor(NettyRemoteClientModule, InetSocketAddress, Option, Function1)
akka.remote.netty.NettyRemoteSupport.withClientFor(InetSocketAddress, Option, Function1)
akka.remote.netty.NettyRemoteClientModule$class.send(NettyRemoteClientModule, Object, Option, Option, InetSocketAddress, long, boolean, ActorRef, Option, ActorType, Option)
akka.remote.netty.NettyRemoteSupport.send(Object, Option, Option, InetSocketAddress, long, boolean, ActorRef, Option, ActorType, Option)
akka.actor.RemoteActorRef.postMessageToMailbox(Object, Option)
akka.actor.ScalaActorRef$class.$bang(ScalaActorRef, Object, Option)
akka.actor.RemoteActorRef.$bang(Object, Option)

With a pretty vanilla configuration, we have some jobs running and they're producing a fair bit of output and sending that over akka-remote to another service which then sends it to our clients:

WorkerService -> FrontEndService -> Clients

The WorkerService is routinely seeing 3-7 threads blocking with the above stack trace while processing stuff in (typically) 14 out of 16 threads in the global dispatcher.

Now, the code for RemoteClient.send starts out like:

  def send[T](

    message: Any,

    senderOption: Option[ActorRef],

    senderFuture: Option[CompletableFuture[T]],

    remoteAddress: InetSocketAddress,

    timeout: Long,

    isOneWay: Boolean,

    actorRef: ActorRef,

    typedActorInfo: Option[Tuple2[String, String]],

    actorType: AkkaActorType): Option[CompletableFuture[T]] = synchronized { // FIXME: find better strategy to prevent race

So, I'm a bit suspicious of that synchronization there.

We haven't altered the akka.remote.client.buffering configs, so retry-message-send-on-failure is still on and capacity is still -1.