Stream metrics 1.5

[mattrjacobs]

Rebased #981 and #986 against master

[mattrjacobs]

(copied from comment in #981)

This refactoring addresses #950. It is a large change, so I want to give the community a chance to look it over/ask questions before I start in on the 1.5.0 RC series (this will be the first commit there, if satisfactory).

The main changes in this PR:

• Instead of command executions writing directly to HystrixRollingNumber/HystrixRollingPercentile data structures, they now write to a Subject that models a stream of command execution events.
• HystrixCommandMetrics now consumes this stream in 4 ways:
  • Cumulative counter of event types
  • Rolling counter of event types
  • Rolling distribution of command latencies
  • Health counts of a circuit
• Sample-based stream of thread pool / queue utilization

This change has not hurt performance when used in Netflix production for ~24 hours. I will publish jmh results here as well. I'll also start linking more Github issues to this ticket as I build on this work.

There should be no visible change in application behavior as a result of this change (see caveats below). However, it enables future work to add new functionality. (like #333)

Caveats (known differences in behavior)

• Maximum concurrency of command is always 0 (HystrixCommandMetrics#getRollingMaxConcurrentExecutions)
• HdrHistogram is used for latency distributions. Percentile calculation is different from HystrixRollingPercentile.
• On application startup, an object pool of Histograms is allocated. This is currently hardcoded to 1000 objects. It's an area I want to explore more deeply

Comments/questions are welcomed and encouraged.

[cloudbees-pull-request-builder]

NetflixOSS » Hystrix » Hystrix-pull-requests #276 FAILURE
Looks like there's a problem with this pull request

[cloudbees-pull-request-builder]

NetflixOSS » Hystrix » Hystrix-pull-requests #278 FAILURE
Looks like there's a problem with this pull request

[cloudbees-pull-request-builder]

NetflixOSS » Hystrix » Hystrix-pull-requests #279 ABORTED

[cloudbees-pull-request-builder]

NetflixOSS » Hystrix » Hystrix-pull-requests #284 FAILURE
Looks like there's a problem with this pull request

[cloudbees-pull-request-builder]

NetflixOSS » Hystrix » Hystrix-pull-requests #285 FAILURE
Looks like there's a problem with this pull request

[cloudbees-pull-request-builder]

NetflixOSS » Hystrix » Hystrix-pull-requests #286 FAILURE
Looks like there's a problem with this pull request

[cloudbees-pull-request-builder]

NetflixOSS » Hystrix » Hystrix-pull-requests #287 SUCCESS
This pull request looks good

[mattrjacobs]

Closed in favor of rebased #1029