file:cmwyJCh3ar4PkqacwqjQWU
The output of lsof on the java process

Could it have to do with netty direct memory buffers used by Netty?

Could it be the Test Conductor leaking unclosed channels?

Can you attach the minimized test?

So the guilty party is really Netty defaults. It creates a huge number of threads.

These are some stats from the test:

The number of "New I/O worker #X" threads is proportional to the number of cores.
My Macbook: 4 cores 36 workers = 8 per core
Scalable1: 24 cores 194 workers = 8.083 per core
a0:  48 cores 386 workers = 8.042 per core

You could be fooled into thinking that netty would create them on demand since they use a CachedThreadPool by default, but the code that use the ThreadPool explicitly creates all the workers up front :(

Will add configuration options for configuring the number of workers in a simliar fashion as the executors in the dispatchers.

With the fix we use 22 worker threads on a0 instead of 386.

Merged to master and cherry picked to release-2.1

Updating tickets (#2659, #2660, #2661, #2666, #2669, #2671, #2672, #2677, #2678, #2680, #2682, #2683, #2688)