👨‍🌾 TestRate.wall_rate_basics, TestRate.ros_rate_basics and TestRate.rate_basics are flaky in nightly_win_rep CI

[Blast545]

Bug report

Required Info:

• Operating System:
  • Buildfarm, Windows Repeated jobs
• Installation type:
  • Source
• Version or commit hash:
  • Rolling
• DDS implementation:
  • N/A
• Client library (if applicable):
  • rclcpp

Information on the issue

Run a nightly_win_rep job, and these tests will sometime fail with a 44% chance

sql_run.sh calculate_flakiness_jobs.sql "rclcpp.TestRate.wall_rate_basics" "40 days"
job_name|last_fail|first_fail|build_count|failure_count|failure_percentage
nightly_win_rep|2023-10-03|2023-09-01|39|17|43.59
nightly_win_deb|2023-09-09|2023-09-09|38|1|2.63

List of builds where it has happened:

rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3098|2023-09-01 04:00:01.454000|windows-container-0be6c6fc
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3101|2023-09-04 04:00:01.562000|windows-container-b2798132
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3104|2023-09-07 04:00:01.204000|windows-container-b2798132
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3105|2023-09-08 04:00:01.718000|windows-container-8f45de3b
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_deb|2845|2023-09-09 04:00:01.402000|windows-container-b2798132
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3106|2023-09-09 04:00:01.413000|windows-container-61f1fae3
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3107|2023-09-10 04:00:01.560000|windows-container-b2798132
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3109|2023-09-11 23:00:01.434000|windows-container-7abc1e20
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3111|2023-09-14 04:00:01.565000|windows-container-4c98a547
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3116|2023-09-19 04:00:01.288000|windows-container-6266cdc1
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3117|2023-09-20 04:00:01.292000|windows-container-4c98a547
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3119|2023-09-22 04:00:01.460000|windows-container-6266cdc1
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3120|2023-09-23 04:00:01.685000|windows-container-4c98a547
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3121|2023-09-24 04:00:01.447000|windows-container-8e52bd77
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3126|2023-09-29 04:00:01.629000|windows-container-6266cdc1
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3128|2023-10-01 04:00:01.642000|windows-container-4c98a547
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3129|2023-10-02 04:00:01.446000|windows-container-4c98a547
rclcpp.TestRate.wall_rate_basics|rclcpp|nightly_win_rep|3130|2023-10-03 04:00:01.432000|windows-container-8e52bd77

Reference builds:
https://ci.ros2.org/view/nightly/job/nightly_win_rep/3130
https://ci.ros2.org/view/nightly/job/nightly_win_rep/3129

Expected behavior

All test passing

Actual behavior

These tests sometimes fail

[Blast545]

This was probably introduced by: #2123

PTAL @clalancette @AlexeyMerzlyakov 🙏

[fujitatomoya]

@clalancette @Blast545 can you check #2329? sorry that i do not have windows... 😓 but this is what i can see from the failure logs. if you help testing this on windows, that would be really appreciated.

[fujitatomoya]

@Blast545 @Crola1702 closing this issue in favor of #2329 (including iron and humble backports).

@AlexeyMerzlyakov really appreciate the help especially for the testing windows!