Updating tickets (#769, #774, #875, #889, #917, #920, #928, #929, #930, #931, #932, #933, #934, #935, #936, #938, #939, #940, #941, #942, #943, #944, #951, #958, #959, #960, #962, #630, #870, #891, #895)

Updating tickets (#87, #620, #644, #679, #750, #752, #753, #754, #764, #875, #876, #929, #938, #939, #940, #941, #942, #943, #944, #953, #954, #977, #983, #987, #996, #630, #643, #725, #892, #893)

If noone's working on this ticket so far (and also tickets #940-942) - I can take care of them. In the past I've had experience in making monitoring services (with Sigar).

Awesome. Thanks a lot.

(In revision:3cee2fc8ec18f6e5aa61e714083b033c5ac21d38) Internal Metrics API. Fixes #939

• Retreives metrics snapshots of the system the node is running on through JMX monitoring MBeans or Hyperic Sigar (is Sigar library is plugged)
• Allows to set metrics alteration monitors that are triggered, when specific conditions are satisfied (e.g., not enough memory left on the node)
• Nodes publish their local metrics to ZNodes
• In order to maintain good performance, metrics manager internally caches snapshots, and refreshes them from time to time from ZooKeeper

Branch: master

Starting this week.

Is the requirement to have Sigar be optionally plugged in (not added in AkkaBuild), and thus you require this to use reflection (quite a lot of it from the cherry-pick pulled in)? What I mean is, do you really want that much reflection in your design? Or is simply adding the Sigar dep all that bad.

Sigar must be optional.

What is sigar used for here? Only systemLoadAverage?

Personally I would not use reflection. In Atmos we didn't use reflection and it's still optional. However, since the reflection code is already implemented, why not start with that and we can revise later. Very isolated change, and I don't think it's much code.

We could also discuss if the dependency should propagate as a transitive dependency or not. I think I vote for letting users add sigar dependency themselves, if they use this feature and sigar is needed for their environment. Sigar might not be in maven central.

Which function should I use to update the new gossip that has Gossip.meta for the detected metric alert conditions?
environment.publishLatestGossip(newGossip) or notifyListeners(newGossip) ?

/**

• Receives metrics alerts and gossips out a flag of indicators if one or more alerts have been detected.

*/
def receiveAlerts(alerts: FailureAlerts): Unit = {
val localGossip = latestGossip
val newGossip = localGossip copy (meta = localGossip.meta ++ alerts.meta)
notifyListeners(newGossip)
}

thanks.

I don't know how you have implemented the full thing, so I'm unsure, but publishLatestGossip will be invoked periodically and is probably the way to publish meta (part of gossip) also.

/Patrik

Here is the latest status that is squashed
https://github.com/helena/akka/commit/61be9d41cbb1f111fae772f5b555fccadd2bf4d5
Branch: wip-939-cluster-metrics-helena-unsquashed
Status and To Do items on the commit msg.

I look fw to give feedback on your work, and I will be able to do that at the end of next week.

/Patrik

Hi Patrik,
I'm cleaning up and testing the metrics and statistical analysis and have a question:

What are the requirements for the sampled metrics, do you prefer JVM-specific heap only? With sigar I can do the current process, which I think would be akka (the akka pid).

With SIGAR I can get system hard limits for cpu, memory, etc as well as the current process cpu, memory, etc.
The process memory stats are what you would see with the ps/top/etc commands, but for a Java process jmx has more detail on the JVM managed heap than sigar of course.

Of course sigar has many stats that the JVM doesn't provide about the system itself.

For memory metrics I think JVM heap is most interesting.


GC rate could also be interesting.


For CPU I think load average from sigar is best. 1, 5 or 15 minutes? Perhaps 5? Alternative is "CPU combined" from sigar. Not for the pid.


Perhaps we should support a few alternatives - configurable.

I can provide more details tomorrow.

/Patrik

Did you see my feedback on https://github.com/helena/akka/commit/61be9d41cbb1f111fae772f5b555fccadd2bf4d5 ?

I think we should talk about how this information is going to be used and how to publish it in best way. I'll start thinking more about that today.

Then we should also discuss exactly what metrics we should grab from sigar, but that feels rather easy to change when we know more about the usage.

Hi Helena,

We had a discussion about the use case for the metrics. The primary use case, and only one to start with, is for load balancing routers. Such a router would use the metrics to send messages to the least loaded nodes. We skip the threshold and alert conditions (anomalies). Collect absolute metrics and aggregate to exponential moving average, which are published to other nodes.

We separate the transport of the metrics from the ordinary versioned Gossip, i.e. don't use Gossip.meta for the metrics. Instead a separate metrics message should be used. We start with the same kind of gossip protocol as the membership Gossip, but we might change to something else later (perhaps a more subscription based model). The metrics gossip can be simplified compared membership gossip, since it should not be versioned, i.e. no conflicts.

The Cluster publish the metrics to the local event bus, and the routers subscribe to the event bus.

The router will use the metrics to weight probability where to send messages. Not only to the best one. It will not perform immediate big changes to the routing, but adjusting in baby steps, to avoid oscillations.

Metrics to collect:
- cpu combined
- number of processors (cores)
- heap used, committed, max
- network IO read/write (max value?)

Note that we skip the physical memory metrics.

Later we will add:
- network latency (via ping-pong messages)
- mailbox sizes
- ...

Can I do anything to help you with these "new" requirements? Would you like to talk about it over Skype? Would you like that I add a skeleton for the metrics gossip part (and you can fill in the details)?

Hi,

I just saw the review on that (now fairly old) squashed branch https://github.com/helena/akka/commit/61be9d41cbb1f111fae772f5b555fccadd2bf4d5. I've commented and made all the changes. That was most definitely unfinished work, vs work in pull-request status yet, thus the commented out work, missing docs in some places.. which are all cleaned up now.

I'll ponder the new requirements today, which sound good. If you can do the routers that will be a huge help. Work has been very busy so progress on this is slower that I'd like.

Let's do a skype chat Tuesday (tomorrow). 4PM your time? My skype id is holly.edelson

Thanks,
Helena

Aggregation of metrics to one EWMA is far more simple, it was getting very inelegant to implement the initial requirements.
Refactoring this now.

Adding latency is excellent. That's something I can easily add when you are ready.

Good, talk to you tomorrow, 4pm CET.
My skype id is patrik.nordwall

/Patrik

You removed the comment? Is the linked commit ready for review. I'll take a look at it and then you can open a pull request.

The metrics aware, adaptive routers should definitely be done as separate tickets, and this can go in first.

You don't need to assign this to me. You can own it until completed. I'll review anyway.

Pull Request

Updating tickets (#939, #940, #1941, #2213, #2214, #2215, #2219, #2222, #2223, #2239, #2240, #2249, #2250, #2252, #2253, #2254, #2256, #2259, #2263, #2264, #2265, #2267, #2270, #2271, #2275, #2277, #2286, #2287, #2289, #2290, #2303, #2304, #2308, #2310, #2311, #2317, #2323, #2331, #2374, #2392, #2405, #2423, #2425, #2440, #2444, #2445, #2453, #2456, #2459, #2473, #2477, #2491, #2495, #2523, #2534, #2541, #2544, #2545, #2549, #2582, #2583, #2589, #2626)

Updating tickets (#939, #940, #1941, #2081, #2126, #2213, #2214, #2215, #2219, #2222, #2223, #2239, #2240, #2249, #2250, #2252, #2253, #2254, #2256, #2259, #2263, #2264, #2265, #2267, #2270, #2271, #2275, #2277, #2286, #2287, #2289, #2290, #2303, #2304, #2308, #2310, #2311, #2317, #2323, #2331, #2374, #2392, #2394, #2405, #2408, #2423, #2424, #2425, #2440, #2444, #2445, #2449, #2453, #2456, #2459, #2461, #2473, #2477, #2485, #2491, #2495, #2498, #2501, #2505, #2515, #2517, #2523, #2534, #2541, #2544, #2545, #2549, #2582, #2583, #2588, #2589, #2598, #2599, #2618, #2623, #2626, #2627, #2630, #2631, #2633, #2634, #2635, #2637, #2638, #2642, #2643, #2646, #2647, #2648, #2649, #2650, #2653, #2655, #2657, #2658)