Skip to content

Commit

Permalink
docs: document retention,grace period for window queries
Browse files Browse the repository at this point in the history
  • Loading branch information
vinothchandar committed Mar 13, 2020
1 parent f08fe81 commit 77276c8
Show file tree
Hide file tree
Showing 3 changed files with 39 additions and 2 deletions.
38 changes: 38 additions & 0 deletions docs-md/concepts/time-and-windows-in-ksqldb-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -365,4 +365,42 @@ SELECT o.order_id, o.total_amount, o.customer_name, s.shipment_id, s.warehouse
For more information on joins, see
[Join Event Streams with ksqlDB](../developer-guide/joins/join-streams-and-tables.md).

### Late Arriving Events

Often times, events that belong to a window can arrive late (e.g slow networks) and a grace period
may be required to ensure the events are accepted into the window. ksqlDB allows configuring this
behavior, for each of the window types above.

For example, to allow events to be accepted upto 2 hours of delay after the window ends,
you might run a query like:

```sql
SELECT orderzip_code, TOPK(order_total, 5) FROM orders
WINDOW TUMBLING (SIZE 1 HOUR, GRACE PERIOD 2 HOURS)
GROUP BY order_zipcode
EMIT CHANGES;
```

Events that arrive later than the grace period, are dropped and not included in the aggregate result.

### Window Retention

ksqlDB supports configuring how many windows in the past to retain for pull and push queries for
each window type. This capability can be very useful for interactive applications that use ksqlDB
as their primary serving data store.

For example, to retain the computed windowed aggregation results for a week,
you might run the following query:

```sql
SELECT regionid, COUNT(*) FROM pageviews
WINDOW HOPPING (SIZE 30 SECONDS, ADVANCE BY 10 SECONDS, RETENTION 7 DAYS, GRACE PERIOD 30 MINUTES)
WHERE UCASE(gender)='FEMALE' AND LCASE (regionid) LIKE '%_6'
GROUP BY regionid
EMIT CHANGES;
```

Note that the specified retention period should be larger than the sum of window size and any grace
period.

Page last revised on: {{ git_revision_date }}
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,7 @@ public void shouldSetupRetentionWhenNonEmpty() {
when(retention.get()).thenReturn(Duration.ofSeconds(10));

// When:
final Materialized<String, GenericRow, StateStore> returned
= MaterializedFactory.create().create(keySerde, rowSerde, OP_NAME, retention);
MaterializedFactory.create().create(keySerde, rowSerde, OP_NAME, retention);

// Then:
verify(retention).isPresent();
Expand Down

0 comments on commit 77276c8

Please sign in to comment.