Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing benchmarks #5212

Closed
huitseeker opened this issue Aug 28, 2022 · 0 comments
Closed

Failing benchmarks #5212

huitseeker opened this issue Aug 28, 2022 · 0 comments
Labels
narwhal Type: Bug Something isn't working

Comments

@huitseeker
Copy link
Contributor

huitseeker commented Aug 28, 2022 •
edited
Loading

Description

The benchmarks (run on CI on every pushed commit, see the script) are failing since MystenLabs/narwhal@babd3d8 (aka MystenLabs/narwhal#738).

This is impossible to reproduce outside Github for me. I had to resort to hacks in order to reproduce this.

This generated the following log collating all the logs from the primaries:
https://gist.github.com/huitseeker/c9e94b03f1ded5155ab6f7fae8197a9c

This shows a lot of ERROR messages:


2022-08-28T00:28:40.938450005Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: WEoJCTeNr8hGZH6h, error type: BatchTimeout
2022-08-28T00:28:41.009256736Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: ZSmM9/zTy0rchjVe, error type: BatchTimeout
2022-08-28T00:28:41.18922286Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: 3tBtWhKDXo0Z1lOt, error type: BatchTimeout
2022-08-28T00:28:41.190253959Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: PWvIQwgrjsBU9zVh, error type: BatchTimeout
2022-08-28T00:28:41.190262359Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: 6cdRBZ0BjSzCKTD/, error type: BatchTimeout
2022-08-28T00:28:41.190270059Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: wZmmum8O9Fji92bQ, error type: BatchTimeout
2022-08-28T00:28:41.190276659Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: GDpuvWoV3imlr11J, error type: BatchTimeout
2022-08-28T00:28:41.233743717Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: T4aZ3TVPYNOquRTy, error type: BatchTimeout
2022-08-28T00:28:43.355264748Z ERROR handle_get_block_command{block_id=kR1FgPrIAvPBgeUI1hlJy36+hIafP2B+6mBCFMGdMG4=}: primary::block_synchronizer::handler: Error for payload synchronization with block id kR1FgPrIAvPBgeUI, error: Payload for block with kR1FgPrIAvPBgeUI couldn't be synchronized: Block with id kR1FgPrIAvPBgeUI could not be retrieved, timeout while retrieving result
2022-08-28T00:28:43.355795947Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: kR1FgPrIAvPBgeUI, error type: BatchError
2022-08-28T00:28:43.355811547Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: ez1A69+h0dSPWy9S, error type: BatchTimeout
2022-08-28T00:28:45.358290095Z ERROR handle_get_block_command{block_id=8fBpctxds6AsKN0wiRB3VVhFgMOkRHNgLbh7pUjYX8Y=}: primary::block_synchronizer::handler: Error for payload synchronization with block id 8fBpctxds6AsKN0w, error: Payload for block with 8fBpctxds6AsKN0w couldn't be synchronized: Block with id 8fBpctxds6AsKN0w could not be retrieved, timeout while retrieving result
2022-08-28T00:28:45.359836393Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: 8fBpctxds6AsKN0w, error type: BatchError
2022-08-28T00:28:45.359848193Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: +SOaQXV+9d25O7Xr, error type: BatchTimeout
2022-08-28T00:28:45.359854893Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: RlIGqn8nBJiuZzsP, error type: BatchTimeout
2022-08-28T00:28:45.359861493Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: 5EZZEkGzUplc5Fbm, error type: BatchTimeout
2022-08-28T00:28:45.359867393Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: ZisR6+IKv3Ox2V9W, error type: BatchTimeout
2022-08-28T00:28:45.359878593Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: oBvDNGqJ24KwLVft, error type: BatchTimeout
2022-08-28T00:28:45.360496692Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: j5w8zYnFTaEfp/Pe, error type: BatchTimeout
2022-08-28T00:28:45.360664692Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: 9AQp+iKRg/9xRJbl, error type: BatchTimeout
2022-08-28T00:28:45.360672392Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: cBPOlHz8S0e0qqZ/, error type: BatchTimeout
2022-08-28T00:28:45.360678192Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: Y/sf8ljv0fjuoxc1, error type: BatchTimeout
2022-08-28T00:28:45.362037691Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: KK2rt+Z4auYyVRCR, error type: BatchTimeout
2022-08-28T00:28:45.362047091Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: 3y26qDLPg5jkZN5R, error type: BatchTimeout
2022-08-28T00:28:45.362053491Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: hMobCuAHhM7ERFMX, error type: BatchTimeout
2022-08-28T00:28:45.362059691Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: 1PZ6pzwpMQ4PvmHG, error type: BatchTimeout
2022-08-28T00:28:45.362065391Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: xASlHPo863vlw4dc, error type: BatchTimeout
2022-08-28T00:28:45.36319459Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: yX+oXs2WkyNsdGxU, error type: BatchTimeout
2022-08-28T00:28:45.36320419Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: 5PPiV86Hvj9NMIXP, error type: BatchTimeout
2022-08-28T00:28:45.36320979Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: LvUmM69H51TG8CNi, error type: BatchTimeout
2022-08-28T00:28:45.36321569Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: Yey4HkTQC7xUGJTq, error type: BatchTimeout
100 2397k  100 2397k    0     0  8789k      0 --:--:-- --:--:-- --:--:-- 8911k
2022-08-28T00:28:47.366147134Z ERROR handle_get_block_command{block_id=t8Bvt5jwALnig7b8j+fIfIdFvdt2TPpyo+FnGOqZRCo=}: primary::block_synchronizer::handler: Error for payload synchronization with block id t8Bvt5jwALnig7b8, error: Payload for block with t8Bvt5jwALnig7b8 couldn't be synchronized: Block with id t8Bvt5jwALnig7b8 could not be retrieved, timeout while retrieving result
2022-08-28T00:28:47.366415633Z ERROR executor::subscriber: Error while retrieving block via block waiter: block id: t8Bvt5jwALnig7b8, error type: BatchError
huitseeker referenced this issue in huitseeker/narwhal Aug 29, 2022
@huitseeker
fix: extend the batch timeout
cc9d430 
This mitigates #858, which shows that on Github, this timeout is nto large enough.
We also suspect this may alleviate issues encountered in Sui, e.g. failures in the
Sui reconfiguration end_to_end test.
@huitseeker huitseeker mentioned this issue Aug 29, 2022
Merged
huitseeker referenced this issue in huitseeker/narwhal Aug 29, 2022
@huitseeker
fix: extend the batch timeout
e8a9725 
This mitigates #858, which shows that on Github, this timeout is nto large enough.
We also suspect this may alleviate issues encountered in Sui, e.g. failures in the
Sui reconfiguration end_to_end test.
huitseeker referenced this issue in huitseeker/narwhal Aug 30, 2022
@huitseeker
fix: extend the batch timeout
a50c257 
This mitigates #858, which shows that on Github, this timeout is nto large enough.
We also suspect this may alleviate issues encountered in Sui, e.g. failures in the
Sui reconfiguration end_to_end test.
huitseeker referenced this issue in huitseeker/narwhal Aug 30, 2022
@huitseeker
fix: extend the batch timeout
2549521 
This mitigates #858, which shows that on Github, this timeout is nto large enough.
We also suspect this may alleviate issues encountered in Sui, e.g. failures in the
Sui reconfiguration end_to_end test.
huitseeker referenced this issue in MystenLabs/narwhal Aug 30, 2022
@huitseeker
fix: extend the batch timeout
e994f67 
This mitigates #858, which shows that on Github, this timeout is nto large enough.
We also suspect this may alleviate issues encountered in Sui, e.g. failures in the
Sui reconfiguration end_to_end test.
This was referenced Sep 6, 2022
Merged
Closed
mwtian referenced this issue in mwtian/sui Sep 30, 2022
@huitseeker
fix: extend the batch timeout
253cb91 
This mitigates MystenLabs/narwhal#858, which shows that on Github, this timeout is nto large enough.
We also suspect this may alleviate issues encountered in Sui, e.g. failures in the
Sui reconfiguration end_to_end test.
@mwtian mwtian transferred this issue from MystenLabs/narwhal Oct 13, 2022
@mwtian mwtian added narwhal Type: Bug Something isn't working labels Oct 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
narwhal Type: Bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert #738 & #842 MystenLabs/narwhal
3 participants
@huitseeker @mwtian @stefan-mysten