test on robots in production

[julijenv]

During the tests on the last iCub that should be out at the end of the month, we had to launch the following test:

ENCODERS TEST

From icubsrv, we typed:
cd $ROBOT_CODE/icub-tests/suites
and we executed the following command:
robottestingframework-testrunner -v -s encoders-icub.xml
NB: we modify the file encoders-icub.xml in order to run one test at a time to get a better view of each part.
here the original file encoders-icub.xml:

<?xml version="1.0" encoding="UTF-8"?>

<suite name="Encoders Test Suite">
    <description> Testing encoders</description>
    <environment>--robotname icub</environment>

    <test type="dll" param="--from optical_encoders_drift_left_arm.ini">  OpticalEncodersDrift </test>
    <test type="dll" param="--from optical_encoders_drift_right_arm.ini"> OpticalEncodersDrift </test>
    <test type="dll" param="--from optical_encoders_drift_left_leg.ini">  OpticalEncodersDrift </test>
    <test type="dll" param="--from optical_encoders_drift_right_leg.ini"> OpticalEncodersDrift </test>
    <test type="dll" param="--from optical_encoders_drift_torso.ini">     OpticalEncodersDrift </test>

    <test type="dll" param="--from motorEncoderConsistency_left_arm.ini">  MotorEncodersConsistency </test>
    <test type="dll" param="--from motorEncoderConsistency_right_arm.ini"> MotorEncodersConsistency </test>
    <test type="dll" param="--from motorEncoderConsistency_left_leg.ini">  MotorEncodersConsistency </test>
    <test type="dll" param="--from motorEncoderConsistency_right_leg.ini"> MotorEncodersConsistency </test>
    <test type="dll" param="--from motorEncoderConsistency_torso.ini">     MotorEncodersConsistency </test>
    <test type="dll" param="--from motorEncodersConsistency_face.ini">     MotorEncodersConsistency </test> 
    <test type="dll" param="--from motorEncodersConsistency_head.ini">     MotorEncodersConsistency </test>
</suite>

Every single test is actually refering to some specific .ini file that can be located into the repository:
cd $ROBOT_CODE/icub-tests/suites/contexts/icub/
Let's look at the following the file optical_encoders_drift_left_arm.ini as example

robot     ${robotname}
part      left_arm
joints    (0 1 2 3)
home      (-30 30 10 45)
speed     (20 20 20 20)
max       (-20 40 20 55)
min       (-40 20 0  35)
cycles    1000
tolerance 1.0 
plot_enabled 0

From what we understood TWEAKING the parameter TOLERANCE could prevent the entire test to FAIL miserably as it was the case until we have been notified to modify it.

QUESTIONS

So here are my questions regarding THIS specific test to be addressed to the creator:

1. is there any documentation for this test?
2. why don't the test check by itself different values for TOLERANCE to compatibility give the final result of [PASS] or [FAIL] with no tweak required and in case of FAILURE, giving some advice on what is the problem or what exactly makes this test fail?
3. Most probably, the second question brings a solution on maybe it has not yet been implemented, so I would be grateful if it was possible to get the range of values for which we can consider the test really passed?
   MIN_VALUE<=TOLERANCE<=MAX_VALUE

JOINTLIMIT TEST

From icubsrv, we typed:
cd $ROBOT_CODE/icub-tests/suites
and we executed the following command:
robottestingframework-testrunner -v -s jointsLimits_robotEth.xml
NB: we modify the file jointsLimits_robotEth.xml in order to run one test at a time to get a better view of each part.
here the original file jointsLimits_robotEth.xml:

<?xml version="1.0" encoding="UTF-8"?>

<suite name="Motor Control Suite">
    <description>Testing robot motors</description>
    <environment>--robotname icub --context icubEth</environment>
    <!--fixture param="--fixture icub-fixture.xml"> yarpmanager </fixture-->

    <!-- Joint Limits -->
    <test type="dll" param="--from joint_limits_right_arm.ini"> JointLimits </test>
    <test type="dll" param="--from joint_limits_left_arm.ini"> JointLimits </test>
    <test type="dll" param="--from joint_limits_right_leg.ini"> JointLimits </test>
    <test type="dll" param="--from joint_limits_left_leg.ini"> JointLimits </test>
    <test type="dll" param="--from joint_limits_torso.ini"> JointLimits </test>
    <test type="dll" param="--from joint_limits_head.ini"> JointLimits </test> 
    <!-- only for robot with mc4plus on forearm use following test 
    <test type="dll" param="--from joint_limits_left_forearm.ini"> JointLimits </test>-->
</suite>

Every single test is actually referring to some specific .ini file that can be located into the repository:
cd $ROBOT_CODE/icub-tests/suites/contexts/icubEth/
Let's look at the following the file joint_limits_head.ini as an example:

robot     ${robotname}
part      head
joints    (0 1 2 3 4 5)
home      (0 0 0 0 0 0)
speed     (20 20 20 20 20 20)
outputLimitPercent (50 50 90 90 90 90)
outOfBoundPosition ( 2  2  2  2  2  2)
tolerance 1 

Here again, the first time we launched the test it FAILS, and therefore we needed to tweak the value in order to make it passed BUT unfortunately we dont know if the value for TOLERANCE is TOLERABLE, an indication on the boundaries of TOLERANCE will be much appreciated.

Except from the TOLERANCE parameter I have to say that the test is really PROBLEMATIC because it makes the robot at risk of breaking some part (mainly cover) as the boundaries for some critical joints are NOT correct. (for example: j1 arms can go below 15 degrees, either from the yarpmotorgui, which is even WORSE in my opinion than the test itself, or by the commands requested by the tests which first get the joint limit from IControlLimits::getLimits() which gets it from the values onto the configuration file:
ex:
cd $ROBOT_CODE/robotd-configuration/iCubHongKong01/hardware/mechanicals/
In the file left_arm-eb1-j0_3-mec.xml, one section of it refers to the hardware limit of the robot:

<group name="LIMITS">
        <!--                                    0                1              2              3            -->
        <param name="hardwareJntPosMax">        8              160             80            106       </param>  
        <param name="hardwareJntPosMin">      -95.5              0            -32             15       </param>
        <param name="rotorPosMin">              0                0              0              0       </param> 
        <param name="rotorPosMax">              0                0              0              0       </param>
    </group>

The problem is for joint1 if the test requests to go to the minimum limit, the arm just BREAKS the torso cover 100% sure and probably also the arm cover. If the test is done WITHOUT COVER I am not 100% sure that this test is safe as well.

j1 of the arm is an example but more joints could be faulty as well

QUESTIONS

Here is my thoughts about this test:
In my opinion, this test is not responsible for the bad numbers, values not correctly inserted into the configurations file of the robot - but if it would have been tested thoroughly it would have made emerged that something was off.
I would love to have someone taking care of double-checking every joint of the robot:

1. the hardware limit of the robot WITHOUT and WITH COVER and modify some files accordingly
2. redefine boundaries for the yarpmotorgui to prevent unexpected collisions if moved carelessly
3. Find out the tolerances for which collisions are avoided and by software prevents them.
   4, 5 and 6 are basically the copy of the questions 1, 2 and 3 from encoders test.

Hope this will benefit and ease the production tests and the external users

cc @pattacini , @randaz81 , @valegagge
fyi @Fabrizio69, @davidetome

[Nicogene]

Hi @julijenv,
honestly I never saw, and I never used those tests, but there are these documentation pages that can be useful.

Encoder test

https://robotology.github.io/icub-tests/doxygen/doc/html/classOpticalEncodersConsistency.html

Joints limits

https://robotology.github.io/icub-tests/doxygen/doc/html/classJointLimits.html

[pattacini]

@Nicogene is not the author of those tests as you can easily check:

• https://github.com/robotology/icub-tests/blame/devel/src/motorEncoders-consistency/motorEncodersConsistency.cpp.
• https://github.com/robotology/icub-tests/blame/devel/src/jointLimits/jointLimits.cpp.

Therefore, we need to deal with this issue according to our internal workflow.

[julijenv]

Dear @Nicogene ,
We unfortunately already look into those documentations but could not find what was really necessary to get a [PASS] test, therefore we created this issue to let the owner add more details on it or find someone to deal with the unexplained and most importantly being able to test and certify that the robot in production is top-notched satisfying all our internal tests.

[julijenv]

this issue has been revisited thx to the help of @randaz81, some modifications to the source code and explanations were given. The test now gives the feedback they have been designed for. We now have the certainty that the reports that we produce makes sense and we know what we are testing.
Nevertheless to get a PASS is still a problem because the 1% tolerance of the test is actually a WCS from the point of view of the creator of this test.
The solution would be certainly to check the PID. Hopefully, in the near future, this will be taken in charge.
When this will be done we could extend the test procedures to its overall starting plan - at the beginning the test procedures were very broad and only some of them have been kept.
It could benefit us greatly to have a robot tested more thoroughly.