v3.2 Mixed and Weighted Load

Overview

Mongoose is able to account the metrics for the configured load job. Load job is defined by an I/O type (Create/Read/Update/Delete/...) and other essential parameters. It's may be neccessary to account the performance metrics while performing a load job consisting of other load jobs (sub-jobs). For example create and read the objects/files simultaneously and account the aggregated metrics.

Limitations

There are a set of parameters which are used by shared load monitor and may not be set for a sub-jobs:

• item-output-file
• load-job-name
• load-limit-rate
• load-limit-time
• load-metrics-threshold
• load-metrics-period
• load-metrics-precondition
• load-queue-size
• storage-driver-remote

The values for these parameters will be taken from the 1st "config" element from the list (if any).

Approach

TODO

Configuration

Mixed Load Types

"type" : "mixed",
"config" : [
    {
        "load" : {
            "type" : "create"
        }
    }, {
        "item" : {
            "input" : {
                "file" : "items2read.csv"
            }
        },
        "load" : {
            "circular" : true,
            "type" : "read"
        }
    }, {
        "item" : {
            "input" : {
                "file" : "items2update.csv"
            }
        },
        "load" : {
            "circular" : true,
            "type" : "update"
        }
    }
]

Create using different users

"type" : "mixed",
"config" : [
    {
        "auth" : {
            "id" : "uid0",
            "secret" : "secret0"
        }
    }, {
        "auth" : {
            "id" : "uid1",
            "secret" : "secret1"
        }
    }, {
        "auth" : {
            "id" : "uid2",
            "secret" : "secret2"
        }
    }
]

Weighted

"type" : "mixed",
"weights" : [
    3,
    2,
    1,
],
"config" : [
    {
        "item" : {
            "output" : {
                "path" : "/bucket0"
            }
        }
    }, {
        "item" : {
            "output" : {
                "path" : "/bucket1"
            }
        }
    }, {
        "item" : {
            "output" : {
                "path" : "/bucket2"
            }
        }
    }
]

Reporting

Standard Output

Periodic metrics are displayed as the table like below:

______________________________________________________________________________________________________________________
 Load | Concur| Driver|       Count       |  Job  |    TP [op/s]    |        |  BW [MB/s]  |Latency [us]|Duration [us]
 Type | rency | Count |-------------------| Time  |-----------------|  Size  |-------------|------------|-------------
      |       |       |   Success  |Failed|  [s]  |  Mean  |  Last  |        | Mean | Last |    Mean    |    Mean     
------|-------|-------|------------|------|-------|--------|--------|--------|------|------|------------|-------------
  READ|    100|      1|    38547921|     0|3880.32|9934.211|9892.048| 36.76GB|9.7013|9.6602|        1029|        1324
CREATE|    100|      1|     9639439|     0|3880.32|2484.186|2471.754| 91.93GB|24.261|24.100|        1440|        1332
----------------------------------------------------------------------------------------------------------------------