Build farmer handoff 2018-04-04

[dhood]

cc @nuclearsandwich

• Expected test failures:

  • Composition tests failing consistently on arm and debug, and some release Composition test failures have gotten persistent #86
    • eProsima has a fix and hopefully will release it soon so our nightlies go back to stable. at that point we should let the team know these test failures are not expected anymore.

• Flakiness:

  • service/parameter tests with fastrtps (common, shows as test failures or timeouts)
  • service/parameter tests with connext (less common, shows as a timeout) wait_for_service not being woken by graph events rmw_connext#280
  • muiltithreaded tests hanging with connext Flaky: multithreaded comms tests hanging with connext on osx #103
  • pendulum_teleop hanging (only runs on connext, linux) Pendulum teleop timeout [connext, linux] #94
    • should be less flaky after Pendulum teleop flakiness reduction demos#232
  • hanging tests with fastrtps Various tests hanging sometimes on linux with fastrtps #98
    • TestFindWeakNodes, test_time_source, test_spin
  • DomainParticipantFactory_impl::create_participant():!create failure creating participant failures on connext (seems can happen in any test): Connext: Automatic participant index failed to initialize #93
    • We should keep investigating/pushing on them for what is causing this now
  • test_utilities access violation: Access violation in new test_utilities test rclcpp#458
  • "performance" tests (action pending discussion in Test of node->now() flaky on OS X rclcpp#457)
    • e.g. rclcpp.TestNode.now, timer/waitset testing

• Buildfarmer tasks

  • reinstall mini3 (@dhood)
  • deploy [configuration] Modify node labels to not use _slave ci#141
  • keep an eye on building the bridge requires a lot of memory #106 and resolve if needed
  • Switch to VS2017 once Connext provides public binaries (use Visual Studio 2017 ci#122)
  • Configure eatable as ros2 node when appropriate, also Move jenkins workspace from C:/ to D:/ on eatable.local #51

• Things to know

  • colcon python packages installed on (some?) windows machines colcon installed on windows machines #105 (not necessarily an issue but should be kept in mind)
  • flake8-builtins is pinned and un-pinning is pending discussion in Flake 8 builtins failures #104
  • I deployed Modify node labels to not use _slave ci#142 and Use --rm on docker run invocation ci#133 yesterday (no issues so far)

[dhood]

forgot to mention rcl's get_node_names is also flaky and failures look like this: https://ci.ros2.org/view/nightly/job/nightly_osx_repeated/775/testReport/junit/(root)/projectroot/test_get_node_names__rmw_fastrtps_cpp/

[dhood]

@nuclearsandwich asked what tools I used last stint to diagnose things as build cop. Mostly it's just being familiar with expected failures so you can spot things out of the ordinary (hence the detailed handoff) but I also found the following useful and added it to the buildcop wiki:

• To get the exact list of code that changed, generate a diff between the output of vcs export --exact between two builds.
  • This is particularly useful for external repos such as FastRTPS where old commits may get pushed to the master branch overnight, and are difficult to spot in the GitHub UI.
• If linter failures occur overnight it is usually because of a new version of a linter.
  • Create a diff of the last pip freeze output of two builds to see which versions changed.
  • If it's a regression in a dependency you can pin the older version temporarily, see this PR for an example.

[nuclearsandwich]

Closing as I've handed off to mjcarroll in #117