Skip to content

Rollup

Jump to bottom Edit New page
Chandra Addala edited this page Jun 15, 2016 · 2 revisions

Rollup Tool

Understanding Rollups is easier once you follow the logic of the rollup operation for a single metric for a single tenant.

Rolling up a single metric for a single tenant between start and end times:

File: RollupTool.java

Inputs expected are:

  • tenant id
  • metric id
  • start timestamp
  • end timestamp

Notes on RollupTool.java:

  1. For each granularity, the rollups are recomputed from the adjacent lower granularity.

  2. RollupTool.java gets 'rollup type' from columnFamily metrics_metadata. Mostly, the 'rollup type' will be BF_BASIC for the simplest case.

    Example:

                        key  | column1     | value
-----------------------------+-------------+------------------------
tg_tenant.example.metric.one | rollup_type | 0x040842465f4241534943
tg_tenant.example.metric.one |        type |               0x040149

  1. For each granularity, RollupTool.java gets the discrete range of time intervals that lie between the specified start and stop times.

  2. For each interval, RollupTool.java gets the data to be rolled up from the previous granularity.

    • For example, if the granularity to rollup is 5m then the previous lower granularity is FULL. Therefore, the source data will be from the 'metrics_full' column family and the destination data will be going into metrics_5m

  3. The rollup is different depending on whether the rollup is rolling up raw data or rollup data

    • If we are computing the 5m rollups, the previous granularity will be FULL. In this case, RollupTool.java will use the RollupComputer class 'Rollup.BasicFromRaw.'

    • If we are rolling up the 20m granularity the previous granularity will be 5m which is itself rollup data. Since we are computing a rollup from previously rolled up data RollupTool.java will use the RollupComputer class 'Rollup.BasicFromBasic.'

    • The difference is demonstrated here:

average.handleFullResMetric(numericMetric.getValue()); ///Rollup from RAW
average.handleRollupMetric(basicRollup); //Rollup from Rollup

Rollup Service

Important Terms

Shards: The metrics are partitioned into shards. ie., each shard is responsible for a set of slots.

Slots: The number of locations to save a metric within a certain time period. This is equal to the number of times a discrete 'granularity' occurs over a given time range. For example:

  • Over a 14 day period, the number of 5-minute blocks is 4032. Hence, the number of 5m slots over a 14-day period is 4032.
    • (14 days * 24 hours * 12 five-minute periods in an hour = 4032)
  • For the 20m granularity over 14 days, the number of slots is 1008.
    • (14 days * 24 hours * 3 twenty-minute periods in an hour = 1008)

If the collectionTime at the 5m granularity falls in the slot 100, after rollups the corresponding collectionTime slot for granularity 20m will be 100/4 = 25. Granularity Code

Setup for Rollups during Ingestion:

During ingestion itself, a shard is assigned to the incoming metric. Following is a snap shot of the Metrics_Locator CF after Ingestion.

 key | column1                             | value
-----+-------------------------------------+-------
  13 | tgtenantrollup.example.metric.three |      
  13 |           tgtest.example.metric.one |      
  96 |   tgtenantrollup.example.metric.two |      
  22 |   tgtenantrollup.example.metric.one |

In this example, the key column contains the shard, column1 contains the metrics locator. For example, tgtenantrollup.example.metric.three has been assigned shard 13, tgtenantrollup.example.metric.two has been assigned the shard 96.

Also during ingestion, the slot for each granularity will also be calculated. In the following example, shard 96 (tgtenantrollup.example.metric.two) is managing slot 6 for the granularity 1440m (1 day), slot 448 for granularity 20m and so on.

The A represents Active state ie rollups have not completed for this Shard-Slot-Granularity tuple. This example is from the Metrics_State cf.

At the end of ingestion, these states of shard and slot are updated in the class ShardStateManager. This is in-memory representation is regularly written to the database, namely the Metrics_State cf by ShardStatePusher.

 key | column1           | value
-----+-------------------+---------------
  96 | metrics_1440m,6,A | 1206509892855
  96 | metrics_20m,448,A | 1206509892855
  96 | metrics_240m,37,A | 1206509892855
  96 | metrics_5m,1795,A | 1206509892855
  96 | metrics_60m,149,A | 1206509892855

Schedule Context is passed to the BatchWriter during ingestion code.

This ScheduleContext (update method) will use the ShardStateManager and SlotState to save which slots need to be rolled up code , ie the slots managed that need to be rolled up are marked dirty in ShardStateManager code.

ShardStatePusher

This is an independent service that is started in the BlueFloodServiceStarter class when Ingestion, Rollup, Query services are started. This service takes the slots marked dirty in ShardStateManager and pushes their state to Cassandra in the Metrics_State cf.

code

ShardStatePuller

Does the exact inverse of the ShardStatePusher. Reads Metrics_State cf and populates the ShardStateManager data structures.

code

RollupService

It has 3 thread pools - LocatorFetch, RollupRead, RollupWrite code

LocatorFetchRunnable: code Retrieves all the locators of the metrics that need to be rolled up for a particular shard from Metrics_Locator cf. The ScheduleContext tells the RollupService which SlotKey (combination of shard, granularity and slot) needs to be rolledup.

The shard is extracted from the slotkey and given to the LocatorFetchRunnable to retrieve all locators. It then creates an ExecutionContext for executing the rollups. code If any exception occurs while executing the rollups, the ExecutionContext is marked successful or unsuccessful. If it is marked unsuccessful, the ScheduleContext is told to schedule the whole batch of rollups again.

RollupRunnable: code This is the runnable that actually performs the work of doing the rollups. The logic of this runnable is very similar to the RollupTool documented above.

SlotStateManager: It manages the slots for a particular Shard. It is a part of ShardStateManager

Quirk for delayed metrics code:

In createOrUpdateForSlotAndMillisecond(), if a metric is ingested for which data has already been rolled up (X), then the rolled up data is again marked Active (A). So that it gets picked up again for rollups by the ScheduleContext.

Total number of Rollups a day:

  • 5m rollups = 288
  • 20m rollups = 72
  • 1h rollups = 24
  • 4h rollups = 6
  • 1d rollups = 1
  • Total = 391

TTL

Different values of TTL are set for each granularity as shown below.

  • Full: TTL of 5 days (unless configured with properties ARE_TTLS_FORCED=true, TTL_CONFIG_CONST=3 in which case its 3 days)
  • 5m: TTL of 10 days
  • 20m: TTL of 20 days
  • 60m: TTL of 155 days
  • 240m: TTL of 10 months approximately
  • 1440m: TTL of 5 years approximately

Note: The TTL configurations are hard coded in CassandraModel.java and SafetyTtlProvider.java. SafetyTtlProvider takes the values set in CassandraModel and multiplies them 5 which result in the values displayed above.

Add a custom footer

Setup

API

Development

General

Our community

Contributing

FAQ

Clone this wiki locally