Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve ReactiveEventPublisher throughtput #1457

Closed
Tracked by #1505
fjtirado opened this issue Aug 31, 2024 · 7 comments · Fixed by apache/incubator-kie-kogito-runtimes#3654 or apache/incubator-kie-kogito-apps#2093
Assignees

Comments

@fjtirado
Copy link

EventPublisher has the list of events generated at the end of a UnitOfWork

These events are currently sent one by one by ReactiveEventMessagePublisher, which is the case of Http Smallrye connector heavily affects system throughput, which in turn increase memory consumption because of event queuing.

One possible improvement is to set the message body to the entire collection, minimizing the number of external calls. However this will affect consumers. For example, in DataIndex, ReactiveMessageEventConsumer will have to be changed to check if the message body contains one event or a collection of them.

@rgolangh
Copy link

rgolangh commented Sep 1, 2024

What is the root cause of the sending being too slow? I remember that the numbers of events in the stress tests were not big.

gmunozfe added a commit to fjtirado/kogito-apps that referenced this issue Sep 18, 2024
gmunozfe added a commit to fjtirado/kogito-apps that referenced this issue Sep 18, 2024
@fjtirado
Copy link
Author

What is the root cause of the sending being too slow? I remember that the numbers of events in the stress tests were not big.

@rgolangh The number of processes was not too big, but the number of events per process is.

gmunozfe added a commit to fjtirado/kogito-apps that referenced this issue Sep 20, 2024
@fjtirado
Copy link
Author

This issue depends on this one quarkiverse/quarkus-reactive-messaging-http#281

fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Sep 26, 2024
fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Sep 26, 2024
fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Sep 26, 2024
fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Sep 26, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Sep 26, 2024
fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Sep 26, 2024
fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Sep 26, 2024
fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Sep 26, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Sep 26, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Sep 27, 2024
fjtirado added a commit to fjtirado/kogito-examples that referenced this issue Sep 27, 2024
fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Sep 27, 2024
@fjtirado fjtirado reopened this Oct 1, 2024
@fjtirado
Copy link
Author

fjtirado commented Oct 1, 2024

@tkobayas
its really scary ;)
anyway, it can be easily fixed if it ever happen again, thanks for the heads up!

fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Oct 1, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Oct 1, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Oct 1, 2024
@tkobayas
Copy link

tkobayas commented Oct 4, 2024

^ @fjtirado Probably because the commit message Fix apache/incubator-kie-issues#1457 contains apache so, it can close the upstream issue even from a fork?

@fjtirado fjtirado reopened this Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-runtimes that referenced this issue Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Oct 4, 2024
fjtirado added a commit to fjtirado/kogito-apps that referenced this issue Oct 4, 2024
fjtirado added a commit to apache/incubator-kie-kogito-runtimes that referenced this issue Oct 4, 2024
* [Fix apache/incubator-kie-issues#1457] Allow grouping of events

* Add test for coverage

* [Fix apache/incubator-kie-issues#1457] Grouping of event serialization

* [Fix apache/incubator-kie-issues#1457] Adding default constructor

* [Fix apache/incubator-kie-issues#1457] Adding TYPE constant

---------

Co-authored-by: gmunozfe <[email protected]>
rgdoliveira pushed a commit to rgdoliveira/kogito-apps that referenced this issue Oct 10, 2024
)

* Add unit tests

* [Fix apache/incubator-kie-issues#1457] Grouping of events 2nd approach

* Add integration tests

* [Fix apache/incubator-kie-issues#1457] Grouping of event serialization

* [Fix apache/incubator-kie-issues#1457] DataAudit serialization

* Update data-index/data-index-service/data-index-service-common/src/test/java/org/kie/kogito/index/service/messaging/AbstractMessagingConsumerIT.java

Co-authored-by: Gonzalo Muñoz <[email protected]>
Update data-index/data-index-service/data-index-service-postgresql/src/test/java/org/kie/kogito/index/service/messaging/PostgreSqlMessagingKafkaConsumerIT.java

Co-authored-by: Gonzalo Muñoz <[email protected]>
Update data-index/data-index-service/data-index-service-common/src/test/java/org/kie/kogito/index/service/messaging/AbstractMessagingConsumerIT.java

Co-authored-by: Gonzalo Muñoz <[email protected]>
Update data-index/data-index-service/data-index-service-postgresql/src/test/java/org/kie/kogito/index/service/messaging/PostgreSqlMessagingKafkaConsumerIT.java

Co-authored-by: Gonzalo Muñoz <[email protected]>
Fix to run collections tests for all

* [Fix apache/incubator-kie-issues#1457] Converter deserialization

* [Fix apache/incubator-kie-issues#1457] Converter deserialization right

---------

Co-authored-by: gmunozfe <[email protected]>
rgdoliveira pushed a commit to rgdoliveira/kogito-runtimes that referenced this issue Oct 10, 2024
* [Fix apache/incubator-kie-issues#1457] Allow grouping of events

* Add test for coverage

* [Fix apache/incubator-kie-issues#1457] Grouping of event serialization

* [Fix apache/incubator-kie-issues#1457] Adding default constructor

* [Fix apache/incubator-kie-issues#1457] Adding TYPE constant

---------

Co-authored-by: gmunozfe <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 🎯 Done
3 participants