#1053
  • Created on
    2011-07-26 00:43
  • Reported by
    bruce.mitchener
  • Status
    Invalid
  • Priority
    Normal (3)
  • Assigned to
    -
  • Milestone
    2.0-M1
  • Plan Level
    -
  • Due Date
    -

Attachments

Related Tickets

Add new subtask or bug

Rockwell: Remote reconnect sometimes has failure in decompression

https://github.com/datafueled/akka-rockwell/tree/master/reconnect-problems demonstrates this ... although you may have to run it a couple of times.

You can end up getting an error like:

RemoteServerClientConnected: akka.remote.netty.NettyRemoteSupport@50b2fb1e, Some(/127.0.0.1:52017)
RemoteServerError: akka.remote.netty.NettyRemoteSupport@50b2fb1e decompression failure (-3): incorrect data check
org.jboss.netty.handler.codec.compression.CompressionException: decompression failure (-3): incorrect data check
at org.jboss.netty.handler.codec.compression.ZlibUtil.exception(ZlibUtil.java:35)
at org.jboss.netty.handler.codec.compression.ZlibUtil.fail(ZlibUtil.java:31)
at org.jboss.netty.handler.codec.compression.ZlibDecoder.decode(ZlibDecoder.java:144)
at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:72)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
User picture
  on 2011-08-02 06:27 *
By Jonas Bonér
Milestone set to 2.0
User picture
  on 2011-08-07 00:41 *
By bruce.mitchener
Bad news.

I can get this with 1.2 snapshot without reconnect turned on, first a closed channel and then the decompression failure:

[ERROR] [2011-08-05 21:47:30.043] [akka:event-driven:dispatcher:global-27] c.d.r.RemoteDebugListener: RemoteServerError: akka.remote.netty.NettyRemoteSupport@560932fe null
java.nio.channels.ClosedChannelException: null
	at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:636) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:369) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.Channels.write(Channels.java:632) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.Channels.write(Channels.java:593) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.handler.codec.compression.ZlibEncoder.finishEncode(ZlibEncoder.java:301) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.handler.codec.compression.ZlibEncoder.handleDownstream(ZlibEncoder.java:237) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:60) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:60) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.handler.execution.ExecutionHandler.handleDownstream(ExecutionHandler.java:167) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.Channels.close(Channels.java:720) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:208) ~[netty-3.2.4.Final.jar:na]
	at akka.remote.netty.RemoteServerHandler.exceptionCaught(NettyRemoteSupport.scala:1011) ~[akka-remote-1.2-SNAPSHOT.jar:na]
	at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:316) ~[netty-3.2.4.Final.jar:na]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_26]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_26]
	at java.lang.Thread.run(Thread.java:680) ~[na:1.6.0_26]
[ERROR] [2011-08-05 21:47:30.046] [akka:event-driven:dispatcher:global-27] c.d.r.RemoteDebugListener: RemoteServerError: akka.remote.netty.NettyRemoteSupport@560932fe decompression failure (-3): incorrect data check
org.jboss.netty.handler.codec.compression.CompressionException: decompression failure (-3): incorrect data check
	at org.jboss.netty.handler.codec.compression.ZlibUtil.exception(ZlibUtil.java:35) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.handler.codec.compression.ZlibUtil.fail(ZlibUtil.java:31) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.handler.codec.compression.ZlibDecoder.decode(ZlibDecoder.java:144) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:72) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280) ~[netty-3.2.4.Final.jar:na]
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200) ~[netty-3.2.4.Final.jar:na]
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_26]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_26]
	at java.lang.Thread.run(Thread.java:680) ~[na:1.6.0_26]
User picture
  on 2011-08-07 01:02 *
By viktorklang
Can you turn compression off?
User picture
  on 2011-08-07 23:41 *
By bruce.mitchener
Well, that is hugely surprising.

I turned off compression and:

  • Random timeouts went away.
  • Average latency went up a bit (no big deal).
  • Random total pauses went away.
  • CPU usage went way up as it started to run smoothly for a change.

And even turning compression back on, I see that I don't get the above stack trace repeatably ... but I do get all of the pain back. So a one line (config) change that changes everything.

I haven't been able to figure out why compression is having such a hugely negative impact.
User picture
  on 2011-10-07 17:42 *
By viktorklang
Status changed from New to Invalid
Compression is removed in 2.0
User picture
  on 2011-12-15 08:51 *
By viktorklang
Milestone changed from 2.0 to 2.0-M1
Updating tickets (#967, #974, #975, #976, #980, #981, #989, #990, #992, #993, #994, #999, #1000, #1004, #1008, #1011, #1015, #1018, #1022, #1023, #1024, #1025, #1027, #1028, #1029, #1030, #1032, #1033, #1036, #1047, #1053, #1062, #1067, #1068, #1069, #1072, #1075, #1078, #1082, #1102, #1107, #1110, #1111, #1115, #1116, #1121, #1122, #1123, #1124)