-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
aandres
committed
Jul 11, 2023
1 parent
520ced1
commit 252b7be
Showing
2 changed files
with
37 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,27 @@ | ||
|
||
# Beavers | ||
|
||
[**Beavers**](https://github.com/tradewelltech/beavers) is a python library for stream processing, optimized for analytics. | ||
[Beavers][1] is a python library for stream processing, optimized for analytics. | ||
|
||
It is used at [Tradewell Technologies][2], | ||
to calculate analytics and serve model predictions, | ||
in both realtime and batch jobs. | ||
|
||
## Key Features | ||
|
||
|
||
* Works in **real time** (eg: reading from kafka) and **replay mode** (eg: reading from parquet) | ||
* Optimized for analytics, it uses micro-batching (instead of processing records one by one) | ||
* Similar to [incremental][3], it updates nodes in a dag incrementally | ||
* Taking inspiration from [kafka streams][4], there are two types of nodes in the dag: | ||
* **Stream:** ephemeral micro-batches of events (cleared after every cycle) | ||
* **State:** durable state derived from streams | ||
* Clear separation between the business logic and the IO. | ||
So the same dag can be used in real time mode, replay mode or can be easily tested. | ||
* Functional interface: no inheritance or decorator required | ||
|
||
|
||
[1]: https://github.com/tradewelltech/beavers | ||
[2]: https://www.tradewelltech.co/ | ||
[3]: https://github.com/janestreet/incremental | ||
[4]: https://www.confluent.io/blog/kafka-streams-tables-part-1-event-streaming/ |