Doesn't gossip from an unreachable node automatically make it reachable again?

No, it has to be DOWNed and then reJOIN.

Okay. So auto-down and auto-rejoin become much more important if the cluster wants to be resilient to transient network issues.

We talked about auto re-join yesterday, and decided to not implement that (now), see #2252.
If auto-down followed by a re-join does the same thing as a manual fresh re-boot would do I think it would be possible to safely implement auto re-join, but using re-join to try to heal network partitions (joining two clusters) is scary business.

Yes, that's I wondered why unreachable nodes can't become reachable again, to allow transient network partitions.

I have only thinking in the terms that Jonas has said, that it must down and join again after becoming unreachable.
The failure detector has configurable acceptable-heartbeat-pause which will allow for short (configurable) transient network partitions.

I wonder what consequences it would have if we continue to send heartbeat to unreachable and that we do the reverse reapUnreachableMembers to detect that an unreachable has become available again (move it from unreachable to members).

I might be overly precocious. I just think that it sounds scary to just let an unreachable node continue as nothing have happened. It might have been down for minutes/hours and the world have moved on. If we allow just continuing then we need to have a timeout window of some sort. The idea was, as Patrik says, to allow the 'acceptable-heartbeat-pause' window to allow transient network partitions. But I am not 100 % sure what is best. WDYT?

So my point is:
1. To allow this we need to have a timeout window (in Terracotta we had this and the default was 1 minute).
2. We already have this timeout window in the 'acceptable-heartbeat-pause' FD option that is there just to allow transient network partitions
3. So, what is the difference in practice? Is it not better to just have a single knob to turn?

I agree. Let's start like that.
For reference, I created a new ticket for auto-restart, see #2273.

Having any unreachable nodes stops the (cluster) world from moving on. Prevents convergence, so no membership or actor partitioning changes are possible. The down action allows the world to move on, and a downed node always goes back through the joining process for this reason. But maybe we just no longer have the original unreachable/down distinction? If a node gets marked unreachable by the failure detector, then it's always on a path to down.

I think the distinction is still valid when using manual admin (auto-down=off). Then it is good that unreachable prevents convergence, so leader actions are not performed until human has decided what to do.

Yes, the distinction is important. Getting back to the beginning of this conversation: if auto-down is off then it could be useful to have an unreachable node that starts to gossip become reachable again. While unreachable the cluster won't change through leader actions, and this allows transient network failures to correct without user intervention. There's probably a big gap for me between thinking about it last year, and changes since, so don't worry if things are different now.

Yep. I agree. That could work. But then I still think it should be a timeout that cuts a line. Reopening ticket until we have discussed this and reached a conclusion.

Discussed again, we rely on acceptable-heartbeat-pause for transient network failures. We can investigate if manual operation of moving unreachable back again is possible.

Updating tickets (#939, #940, #1941, #2213, #2214, #2215, #2219, #2222, #2223, #2239, #2240, #2249, #2250, #2252, #2253, #2254, #2256, #2259, #2263, #2264, #2265, #2267, #2270, #2271, #2275, #2277, #2286, #2287, #2289, #2290, #2303, #2304, #2308, #2310, #2311, #2317, #2323, #2331, #2374, #2392, #2405, #2423, #2425, #2440, #2444, #2445, #2453, #2456, #2459, #2473, #2477, #2491, #2495, #2523, #2534, #2541, #2544, #2545, #2549, #2582, #2583, #2589, #2626)

Updating tickets (#939, #940, #1941, #2081, #2126, #2213, #2214, #2215, #2219, #2222, #2223, #2239, #2240, #2249, #2250, #2252, #2253, #2254, #2256, #2259, #2263, #2264, #2265, #2267, #2270, #2271, #2275, #2277, #2286, #2287, #2289, #2290, #2303, #2304, #2308, #2310, #2311, #2317, #2323, #2331, #2374, #2392, #2394, #2405, #2408, #2423, #2424, #2425, #2440, #2444, #2445, #2449, #2453, #2456, #2459, #2461, #2473, #2477, #2485, #2491, #2495, #2498, #2501, #2505, #2515, #2517, #2523, #2534, #2541, #2544, #2545, #2549, #2582, #2583, #2588, #2589, #2598, #2599, #2618, #2623, #2626, #2627, #2630, #2631, #2633, #2634, #2635, #2637, #2638, #2642, #2643, #2646, #2647, #2648, #2649, #2650, #2653, #2655, #2657, #2658)