Skip to content

v3.5 Monitoring API

Jump to bottom
Andrey Kurilov edited this page Sep 13, 2017 · 10 revisions

Contents

  1. Introduction
  2. Requirements
  3. Limitations
  4. Configuration
    4.1. JMX
    4.2. Prometheus
  5. Reporting
    5.1. JMX
    5.2. Prometheus
    5.3. Standard Output
    5.4. Log Files

1. Introduction

Mongoose computes the rich set of the metrics characterizing each load step been performed. It means that the tool may work as a metrics providing service which may be used to build other applications on top of it. The metrics consumer may be a Mongoose-specific UI application either a monitoring infrastructure system.

2. Requirements

To export the metrics via Prometheus exchange format it's necessary to deploy the standalone JMX Prometheus Exporter which can be downloaded using this link (for the version 0.9 which can change later).

3. Limitations

TODO

4. Configuration

4.1. JMX

To enable the remote monitoring add the following JVM arguments to the command line:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.rmi.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

4.2. Prometheus

To enable the metrics exporting for Prometheus run the standalone JMX Exporter:

java -jar jmx_prometheus_httpserver-0.9-jar-with-dependencies.jar 9280 <MONGOOSE_DIR>/config/jmx_exporter.yaml

Note:

  • Mongoose provides the configuration file for the JMX Exporter.
  • JMX Exporter will try to connect to Mongoose using the 127.0.0.1:9010 address by default. To change this behavior modify the <MONGOOSE_DIR>/config/prometheus_jmx_exporter.yaml configuration file.

5. Reporting

5.1. JMX

  • JMX bean domain name: com.emc.mongoose.load.monitor
  • JMX bean name: test step id
  • JMX bean attributes: see below

Attributes reflect the current metric values for each test step:

  1. ActualConcurrencyMean
  2. ActualConcurrencyLast
  3. ByteCount
  4. ByteRateLast
  5. ByteRateMean
  6. DurationHiQ
  7. DurationLoQ
  8. DurationMax
  9. DurationMean
  10. DurationMed
  11. DurationMin
  12. DurationSum
  13. ElapsedTimeMillis
  14. FailCount
  15. FailRateLast
  16. FailRateMean
  17. LatencyHiQ
  18. LatencyLoQ
  19. LatencyMax
  20. LatencyMean
  21. LatencyMed
  22. LatencyMin
  23. LatencySum
  24. StartTimeMillis
  25. SuccCount
  26. SuccRateLast
  27. SuccRateMean

Note:

Attributes with names ending with "Millis" provide the time values measured in milliseconds. Other attributes providing the time values (DurationXXX and LatencyXXX) are in microseconds.

Example: monitoring the metrics using VisualVM application for the test step with id "step0":

visualvm metrics monitoring screenshot

5.2. Prometheus

  • Metric name pattern: "mongoose_<METRIC_NAME_IN_SNAKE_CASE>"
  • Labels:
    • "load_type": the operation type (NOOP/CREATE/READ/UPDATE/DELETE)
    • "storage_driver_concurrency": the configured concurrency (simultaneously active I/O channels)
    • "storage_driver_count": the configured count of the drivers used to load the storage
    • "test_step_id": the step id which may contain additional information (node IPs, zone ids, whatever) if set by user

The example of JMX Exporter metrics response follows:

# HELP mongoose_duration_med Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>DurationMed)
# TYPE mongoose_duration_med gauge
mongoose_duration_med{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 3.0

# HELP mongoose_elapsed_time_millis Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>ElapsedTimeMillis)
# TYPE mongoose_elapsed_time_millis counter
mongoose_elapsed_time_millis{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 584178.0

# HELP mongoose_duration_sum Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>DurationSum)
# TYPE mongoose_duration_sum counter
mongoose_duration_sum{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 75795.0

# HELP mongoose_byte_count Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>ByteCount)
# TYPE mongoose_byte_count counter
mongoose_byte_count{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 6.15719632896E11

# HELP mongoose_latency_mean Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>LatencyMean)
# TYPE mongoose_latency_mean gauge
mongoose_latency_mean{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 12.196644097733294

# HELP mongoose_duration_hi_q Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>DurationHiQ)
# TYPE mongoose_duration_hi_q gauge
mongoose_duration_hi_q{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 4.0

# HELP mongoose_duration_lo_q Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>DurationLoQ)
# TYPE mongoose_duration_lo_q gauge
mongoose_duration_lo_q{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 2.0

# HELP mongoose_fail_count Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>FailCount)
# TYPE mongoose_fail_count counter
mongoose_fail_count{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 0.0

# HELP mongoose_fail_rate_mean Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>FailRateMean)
# TYPE mongoose_fail_rate_mean gauge
mongoose_fail_rate_mean{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 0.0

# HELP mongoose_duration_min Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>DurationMin)
# TYPE mongoose_duration_min gauge
mongoose_duration_min{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 2.0

# HELP mongoose_succ_rate_last Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>SuccRateLast)
# TYPE mongoose_succ_rate_last gauge
mongoose_succ_rate_last{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 998.299028090015

# HELP mongoose_byte_rate_mean Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>ByteRateMean)
# TYPE mongoose_byte_rate_mean gauge
mongoose_byte_rate_mean{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 1.0540506928553455E9

# HELP mongoose_succ_rate_mean Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>SuccRateMean)
# TYPE mongoose_succ_rate_mean gauge
mongoose_succ_rate_mean{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 1005.221038216288

# HELP mongoose_latency_med Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>LatencyMed)
# TYPE mongoose_latency_med gauge
mongoose_latency_med{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 1.0

# HELP mongoose_latency_sum Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>LatencySum)
# TYPE mongoose_latency_sum counter
mongoose_latency_sum{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 75795.0

# HELP mongoose_latency_lo_q Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>LatencyLoQ)
# TYPE mongoose_latency_lo_q gauge
mongoose_latency_lo_q{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 1.0

# HELP mongoose_duration_max Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>DurationMax)
# TYPE mongoose_duration_max gauge
mongoose_duration_max{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 10695.0

# HELP mongoose_duration_mean Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>DurationMean)
# TYPE mongoose_duration_mean gauge
mongoose_duration_mean{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 22.312334412717103

# HELP mongoose_succ_count Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>SuccCount)
# TYPE mongoose_succ_count counter
mongoose_succ_count{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 587196.0

# HELP mongoose_fail_rate_last Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>FailRateLast)
# TYPE mongoose_fail_rate_last gauge
mongoose_fail_rate_last{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 0.0

# HELP mongoose_latency_hi_q Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>LatencyHiQ)
# TYPE mongoose_latency_hi_q gauge
mongoose_latency_hi_q{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 1.0

# HELP mongoose_latency_max Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>LatencyMax)
# TYPE mongoose_latency_max gauge
mongoose_latency_max{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 10693.0

# HELP mongoose_start_time_millis Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>StartTimeMillis)
# TYPE mongoose_start_time_millis gauge
mongoose_start_time_millis{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 1.499899456106E12

# HELP mongoose_byte_rate_last Attribute exposed for management (com.emc.mongoose.model.metrics<storageDriverConcurrency=1, storageDriverCount=1, loadType=CREATE, stepId=load-20170712.224415.577-1373419525><>ByteRateLast)
# TYPE mongoose_byte_rate_last gauge
mongoose_byte_rate_last{load_type="CREATE",storage_driver_concurrency="1",storage_driver_count="1",test_step_id="load-20170712.224415.577-1373419525",} 1.0467924016785156E9

Note that both counter and gauge metric types are present.

5.3. Standard Output

TODO

5.4. Log Files

TODO

About

Documentation

Development

Notes

Archive

Links

Clone this wiki locally